[LU-4743] soft lockup in lustre 2.4.2

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.6.0, Lustre 2.5.2
• Affects Version/s: Lustre 2.4.2
• Labels: mn4 sdsc
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Minh Diep

Description

After upgraded servers and clients from 1.8.9 to 2.4.2, mounting clients hit lockup.

[root@puma-mds-10-6 etc]# dmesg
LustreError: 3459:0:(osp_sync.c:611:osp_sync_process_record()) unknown record type: 10612401
LustreError: 3459:0:(osp_sync.c:611:osp_sync_process_record()) Skipped 289773766 previous similar messages
LustreError: 3459:0:(osp_sync.c:780:osp_sync_process_queues()) can't send: -22
LustreError: 3459:0:(osp_sync.c:780:osp_sync_process_queues()) Skipped 289773767 previous similar messages
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffff8127f8bc>] [<ffffffff8127f8bc>] number+0x13c/0x320
RSP: 0018:ffff880620641700 EFLAGS: 00000246
RAX: 000000000000002d RBX: ffff8806206417c0 RCX: 00000000fffffffb
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000001
RBP: ffffffff8100bb8e R08: 0000000000000000 R09: 00000000fffffffc
R10: 0000000000000002 R11: 0000000000000001 R12: 00000000ffffffff
R13: 0000000000000001 R14: ffff8806206416e0 R15: ffffffff8100bd6e
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
ffff880620641720 ffff880326cbc000 ffff8806206417c0 ffff880620641740
<d> ffff880600000000 fffffffffffffffe ffff8806ffffffff 000000002d00bd6e
<d> 3130363434303232 0000000000000001 0000000000000001 00000000fffffffd
Call Trace:
[<ffffffff81281990>] ? vsnprintf+0x450/0x5e0
[<ffffffffa031627b>] ? cfs_set_ptldebug_header+0x2b/0xc0 [libcfs]
[<ffffffffa03269ca>] ? libcfs_debug_vmsg2+0x2ea/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: 0d 49 39 dc 0f 82 5f 01 00 00 49 83 c4 01 45 85 c0 74 18 49 39 dc 73 05 41 c6 04 24 30 49 83 c4 01 41 83 ff 10 0f 84 5e 01 00 00 <8b> 85 7c ff ff ff 41 89 c8 85 c0 75 55 41 83 e6 01 41 83 fe 01
Call Trace:
[<ffffffff81281990>] ? vsnprintf+0x450/0x5e0
[<ffffffffa031627b>] ? cfs_set_ptldebug_header+0x2b/0xc0 [libcfs]
[<ffffffffa03269ca>] ? libcfs_debug_vmsg2+0x2ea/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffff8127fb60>] [<ffffffff8127fb60>] format_decode+0x0/0x3b0
RSP: 0018:ffff8806206417c8 EFLAGS: 00000206
RAX: 000000000000000a RBX: ffff880620641860 RCX: 0000000000000032
RDX: 0000000000000000 RSI: ffff880620641810 RDI: ffffffffa0dd0d34
RBP: ffffffff8100bb8e R08: 00000000fffffff9 R09: 00000000fffffffc
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8806ffffffff
R13: fffffffffffffffe R14: ffff880600000000 R15: ffff880620641740
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
ffffffff812815c4 0000000200000010 0000000affffffff ffffffffffffffff
<d> 0000000000000000 000000000000000c 27793632810a15fa 00000000000009ce
<d> ffff880327793632 0000000200000010 0000000affffffff ffffffffffffffff
Call Trace:
[<ffffffff812815c4>] ? vsnprintf+0x84/0x5e0
[<ffffffffa031627b>] ? cfs_set_ptldebug_header+0x2b/0xc0 [libcfs]
[<ffffffffa03269ca>] ? libcfs_debug_vmsg2+0x2ea/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bd6e>] ? reschedule_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc337f>] ? osp_sync_process_queues+0x4ef/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: 41 5c 41 5d 41 5e 41 5f c9 c3 90 c6 03 2e 49 83 c4 01 48 83 c3 01 4c 3b 65 b0 0f 85 78 ff ff ff eb d3 66 0f 1f 84 00 00 00 00 00 <8b> 06 55 48 89 e5 83 f8 01 74 4d 83 f8 02 0f 84 24 01 00 00 c7
Call Trace:
[<ffffffff812815c4>] ? vsnprintf+0x84/0x5e0
[<ffffffffa031627b>] ? cfs_set_ptldebug_header+0x2b/0xc0 [libcfs]
[<ffffffffa03269ca>] ? libcfs_debug_vmsg2+0x2ea/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bd6e>] ? reschedule_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc337f>] ? osp_sync_process_queues+0x4ef/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffff8127f030>] [<ffffffff8127f030>] strlen+0x10/0x30
RSP: 0018:ffff880620641860 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffff880620641860 RCX: 0000000000000000
RDX: ffffffffa0dcf97f RSI: 00000000531d65b8 RDI: ffffffffa0dcf970
RBP: ffffffff8100bb8e R08: 00000000fffffff9 R09: 00000000fffffffc
R10: 0000000000000001 R11: 0000000000000001 R12: 0000000200000010
R13: ffffffff810a1507 R14: ffff8806206417f0 R15: ffffffff81012c59
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
ffff8806206419d0 ffffffffa0326832 0000000000000000 ffff880620641900
<d> ffff880620641960 ffffffff8105e513 ffff880620641fd8 ffff880620608080
<d> ffffffffa0de05c0 0000000000000053 ffff880335cb0010 0002000000000000
Call Trace:
[<ffffffffa0326832>] ? libcfs_debug_vmsg2+0x152/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffff81510ee2>] ? _spin_lock+0x12/0x30
[<ffffffffa0dc337f>] ? osp_sync_process_queues+0x4ef/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: 0f 1f 00 48 83 c0 01 0f b6 10 f6 82 40 14 af 81 20 75 f0 c9 c3 66 0f 1f 44 00 00 31 c0 80 3f 00 55 48 89 fa 48 89 e5 74 11 66 90 <48> 83 c2 01 80 3a 00 75 f7 48 89 d0 48 29 f8 c9 c3 66 66 66 66
Call Trace:
[<ffffffffa0326832>] ? libcfs_debug_vmsg2+0x152/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffff81510ee2>] ? _spin_lock+0x12/0x30
[<ffffffffa0dc337f>] ? osp_sync_process_queues+0x4ef/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffffa032683d>] [<ffffffffa032683d>] libcfs_debug_vmsg2+0x15d/0xbb0 [libcfs]
RSP: 0018:ffff880620641870 EFLAGS: 00000206
RAX: 0000000000000023 RBX: ffff8806206419d0 RCX: 0000000000000000
RDX: ffffffffa0dcf9a7 RSI: 00000000531d660c RDI: ffffffffa0dcf990
RBP: ffffffff8100bb8e R08: 000000000000ac3e R09: 0000000000000000
R10: 0000000000000007 R11: 0000000000000001 R12: ffffffffa0de05c0
R13: ffff880620641830 R14: 000000200000000f R15: ffffffff810a1507
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
0000000000000000 ffff880620641900 ffff880620641960 ffffffff8105e513
<d> ffff880620641fd8 ffff880620608080 ffffffffa0de0640 0000000000000053
<d> ffff880335cb0010 0008000000000000 ffffffffa0dd37e8 0000000000000000
Call Trace:
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: ff e8 08 88 f5 e0 49 8b 7f 08 49 89 c4 8d 40 01 89 85 24 ff ff ff 48 85 ff 74 10 e8 ee 87 f5 e0 41 8d 44 04 02 89 85 24 ff ff ff <8b> 85 24 ff ff ff 44 8b 15 2a 23 02 00 41 bc 55 00 00 00 c7 85
Call Trace:
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffff8127fbe3>] [<ffffffff8127fbe3>] format_decode+0x83/0x3b0
RSP: 0018:ffff8806206417c0 EFLAGS: 00000216
RAX: ffffffffa0dd3809 RBX: ffff8806206417c0 RCX: 0000000000000000
RDX: 0000000000000075 RSI: ffff880620641810 RDI: ffffffffa0dd3808
RBP: ffffffff8100bb8e R08: 0000000000000075 R09: 00000000fffffffa
R10: 0000000000000004 R11: 0000000000000001 R12: 00000000ffffffff
R13: 00000000ffffffff R14: 0000000000000000 R15: ffff880620641740
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
ffff880620641860 ffffffff812815c4 000000000000000f 0000000affffffff
<d> ffffffffffffffff 0000000000000000 0000000000000005 26cc7b0b810a15fa
<d> 00000000000004f5 ffff880326cc7b0b 0000000000000000 0000000affffffff
Call Trace:
[<ffffffff812815c4>] ? vsnprintf+0x84/0x5e0
[<ffffffffa031627b>] ? cfs_set_ptldebug_header+0x2b/0xc0 [libcfs]
[<ffffffffa03269ca>] ? libcfs_debug_vmsg2+0x2ea/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: 8b 46 08 85 c0 0f 88 4d 02 00 00 c7 06 00 00 00 00 48 89 f8 c7 46 10 ff ff ff ff 80 38 2e 0f 84 e4 01 00 00 c7 46 14 ff ff ff ff <0f> b6 10 80 fa 68 74 15 89 d1 83 c9 20 80 f9 6c 74 0b 80 f9 7a
Call Trace:
[<ffffffff812815c4>] ? vsnprintf+0x84/0x5e0
[<ffffffffa031627b>] ? cfs_set_ptldebug_header+0x2b/0xc0 [libcfs]
[<ffffffffa03269ca>] ? libcfs_debug_vmsg2+0x2ea/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffff81510efa>] [<ffffffff81510efa>] _spin_lock+0x2a/0x30
RSP: 0018:ffff880620641840 EFLAGS: 00000246
RAX: 00000000000092f6 RBX: ffff880620641840 RCX: 0000000000000000
RDX: 00000000000092f6 RSI: 0000000000000000 RDI: ffff880335cb0000
RBP: ffffffff8100bb8e R08: 0000000000000000 R09: 00000000fffffffc
R10: 0000000000000001 R11: 0000000000000001 R12: ffff880620641950
R13: ffffffffa0dd0d32 R14: ffffffffa0dd0d34 R15: ffff8803363b205f
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
ffff880620641860 ffffffffa03163ef ffff880335cb0000 0000000000000055
<d> ffff8806206419d0 ffffffffa03267bc 0000000000000000 ffff880620641900
<d> ffff880620641960 ffffffff8105e513 ffff880620641fd8 ffff880620608080
Call Trace:
[<ffffffffa03163ef>] ? cfs_trace_lock_tcd+0x3f/0xa0 [libcfs]
[<ffffffffa03267bc>] ? libcfs_debug_vmsg2+0xdc/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffffa0326bfa>] ? libcfs_debug_vmsg2+0x51a/0xbb0 [libcfs]
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc3276>] ? osp_sync_process_queues+0x3e6/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df <c9> c3 0f 1f 40 00 55 48 89 e5 0f 1f 44 00 00 f0 81 2f 00 00 00
Call Trace:
[<ffffffffa03163ef>] ? cfs_trace_lock_tcd+0x3f/0xa0 [libcfs]
[<ffffffffa03267bc>] ? libcfs_debug_vmsg2+0xdc/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffffa0326bfa>] ? libcfs_debug_vmsg2+0x51a/0xbb0 [libcfs]
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0dc3276>] ? osp_sync_process_queues+0x3e6/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffff8127ef10>] [<ffffffff8127ef10>] strrchr+0x10/0x40
RSP: 0018:ffff880620641860 EFLAGS: 00000206
RAX: ffffffffa0dd33df RBX: ffff880620641860 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000002f RDI: ffffffffa0dd3360
RBP: ffffffff8100bb8e R08: 000000000000ac3e R09: 0000000000000000
R10: 0000000000000007 R11: 0000000000000001 R12: 000000200000000f
R13: ffffffff81281876 R14: ffff880620641860 R15: ffff8803275b1000
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
ffff8806206419d0 ffffffffa032677a 0000000000000000 ffff880620641900
<d> ffff880620641960 ffffffff8105e513 ffff880620641fd8 ffff880620608080
<d> ffffffffa0de0640 0000000000000053 ffff880335cb0010 0008000000000000
Call Trace:
[<ffffffffa032677a>] ? libcfs_debug_vmsg2+0x9a/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0327290>] ? libcfs_debug_msg+0x0/0x50 [libcfs]
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: 00 00 00 48 83 c0 01 0f b6 10 40 38 f2 74 06 84 d2 75 f0 31 c0 c9 c3 0f 1f 40 00 31 d2 80 3f 00 55 48 89 f8 48 89 e5 74 11 66 90 <48> 83 c0 01 80 38 00 75 f7 48 89 c2 48 29 fa 48 8d 04 17 0f 1f
Call Trace:
[<ffffffffa032677a>] ? libcfs_debug_vmsg2+0x9a/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0327290>] ? libcfs_debug_msg+0x0/0x50 [libcfs]
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
LustreError: 3459:0:(osp_sync.c:780:osp_sync_process_queues()) can't send: -22
LustreError: 3459:0:(osp_sync.c:780:osp_sync_process_queues()) Skipped 289922854 previous similar messages
LustreError: 3459:0:(osp_sync.c:611:osp_sync_process_record()) unknown record type: 10612401
LustreError: 3459:0:(osp_sync.c:611:osp_sync_process_record()) Skipped 289922855 previous similar messages
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffffa031602b>] [<ffffffffa031602b>] cfs_trace_buf_idx_get+0x2b/0x30 [libcfs]
RSP: 0018:ffff880620641860 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff880620641860 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000002f RDI: ffffffffa0dd3360
RBP: ffffffff8100bb8e R08: 0000000000000000 R09: 00000000fffffffc
R10: 0000000000000001 R11: 0000000000000001 R12: 0000000200000010
R13: ffffffff81281876 R14: ffff880620641860 R15: ffff880327d4c000
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
ffff8806206419d0 ffffffffa032678a 0000000000000000 ffff880620641900
<d> ffff880620641960 ffffffff8105e513 ffff880620641fd8 ffff880620608080
<d> ffffffffa0de0500 0000000000000053 ffff880335cb0010 0002000000000000
Call Trace:
[<ffffffffa032678a>] ? libcfs_debug_vmsg2+0xaa/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffff81510ee2>] ? _spin_lock+0x12/0x30
[<ffffffffa0dc3276>] ? osp_sync_process_queues+0x3e6/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: 55 48 89 e5 0f 1f 44 00 00 65 48 8b 04 25 c8 cb 00 00 48 63 90 44 e0 ff ff b8 02 00 00 00 f7 c2 00 00 ff 03 75 08 31 c0 f6 c6 ff <0f> 95 c0 c9 c3 55 48 89 e5 0f 1f 44 00 00 0f b7 47 62 66 3b 46
Call Trace:
[<ffffffffa032678a>] ? libcfs_debug_vmsg2+0xaa/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffff81510ee2>] ? _spin_lock+0x12/0x30
[<ffffffffa0dc3276>] ? osp_sync_process_queues+0x3e6/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffff8127ef17>] [<ffffffff8127ef17>] strrchr+0x17/0x40
RSP: 0018:ffff880620641860 EFLAGS: 00000206
RAX: ffffffffa0dd33a2 RBX: ffff880620641860 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000002f RDI: ffffffffa0dd3360
RBP: ffffffff8100bb8e R08: 000000000000ac3e R09: 0000000000000000
R10: 0000000000000007 R11: 0000000000000001 R12: 000000200000000f
R13: ffffffff81281876 R14: ffff880620641860 R15: ffff880327deb000
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
ffff8806206419d0 ffffffffa032677a 0000000000000000 ffff880620641900
<d> ffff880620641960 ffffffff8105e513 ffff880620641fd8 ffff880620608080
<d> ffffffffa0de0640 0000000000000053 ffff880335cb0010 0008000000000000
Call Trace:
[<ffffffffa032677a>] ? libcfs_debug_vmsg2+0x9a/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffff81510ee2>] ? _spin_lock+0x12/0x30
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: 0f b6 10 40 38 f2 74 06 84 d2 75 f0 31 c0 c9 c3 0f 1f 40 00 31 d2 80 3f 00 55 48 89 f8 48 89 e5 74 11 66 90 48 83 c0 01 80 38 00 <75> f7 48 89 c2 48 29 fa 48 8d 04 17 0f 1f 44 00 00 40 38 30 74
Call Trace:
[<ffffffffa032677a>] ? libcfs_debug_vmsg2+0x9a/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffff81510ee2>] ? _spin_lock+0x12/0x30
[<ffffffffa0dc3315>] ? osp_sync_process_queues+0x485/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [osp-syn-28:3459]
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 0
Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) autofs4 sunrpc bonding 8021q garp stp llc ipv6 iptable_filter ip_tables tcp_htcp microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core raid10 myri10ge ses enclosure sg ixgbe mdio igb dca ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 3459, comm: osp-syn-28 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Supermicro X8DTS/X8DTS
RIP: 0010:[<ffffffff812815e0>] [<ffffffff812815e0>] vsnprintf+0xa0/0x5e0
RSP: 0018:ffff8806206417d0 EFLAGS: 00000297
RAX: 0000000000000007 RBX: ffff880620641860 RCX: 0000000000000002
RDX: 000000000000000f RSI: ffff880620641810 RDI: ffffffffa0dd0d0b
RBP: ffffffff8100bb8e R08: 0000000000000078 R09: 00000000fffffffc
R10: 0000000000000001 R11: 0000000000000001 R12: 3130363434303232
R13: 000000002d00bb8e R14: ffff8806ffffffff R15: fffffffffffffffe
FS: 00007feb174b1700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa0315fde CR3: 0000000001a85000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process osp-syn-28 (pid: 3459, threadinfo ffff880620640000, task ffff880620608080)
Stack:
0000000200000010 ffff880620641830 ffffffffa0de0640 0000000000000000
<d> 0000000000000015 27fd94bb810a15fa 0000000000000b45 ffff880327fd94bb
<d> 000000200000000f 00000010ffffffff ffffffffffffffff ffffffffa031627b
Call Trace:
[<ffffffffa031627b>] ? cfs_set_ptldebug_header+0x2b/0xc0 [libcfs]
[<ffffffffa03269ca>] ? libcfs_debug_vmsg2+0x2ea/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffffa03266ea>] ? libcfs_debug_vmsg2+0xa/0xbb0 [libcfs]
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffff81510ee2>] ? _spin_lock+0x12/0x30
[<ffffffffa0dc3276>] ? osp_sync_process_queues+0x3e6/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: 00 00 00 48 8d 75 b0 4c 89 ef e8 9c e5 ff ff 48 63 c8 8b 45 b0 4d 8d 64 0d 00 83 f8 11 76 2c 83 e8 08 83 f8 0b 0f 86 f8 02 00 00 <41> 8b 16 83 fa 2f 0f 86 e7 04 00 00 49 8b 46 08 48 8d 50 08 49
Call Trace:
[<ffffffffa031627b>] ? cfs_set_ptldebug_header+0x2b/0xc0 [libcfs]
[<ffffffffa03269ca>] ? libcfs_debug_vmsg2+0x2ea/0xbb0 [libcfs]
[<ffffffff8105e513>] ? perf_event_task_sched_out+0x33/0x80
[<ffffffffa03266ea>] ? libcfs_debug_vmsg2+0xa/0xbb0 [libcfs]
[<ffffffffa03272d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffff81510ee2>] ? _spin_lock+0x12/0x30
[<ffffffffa0dc3276>] ? osp_sync_process_queues+0x3e6/0x15e0 [osp]
[<ffffffffa0ba4d2b>] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa041235a>] ? llog_cat_process_cb+0x56a/0x620 [obdclass]
[<ffffffffa040de5b>] ? llog_process_thread+0x8fb/0xe00 [obdclass]
[<ffffffffa0411df0>] ? llog_cat_process_cb+0x0/0x620 [obdclass]
[<ffffffffa040fb6d>] ? llog_process_or_fork+0x12d/0x660 [obdclass]
[<ffffffffa0410c49>] ? llog_cat_process_or_fork+0x89/0x350 [obdclass]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0dc2e90>] ? osp_sync_process_queues+0x0/0x15e0 [osp]
[<ffffffffa0410f29>] ? llog_cat_process+0x19/0x20 [obdclass]
[<ffffffffa031775a>] ? cfs_waitq_signal+0x1a/0x20 [libcfs]
[<ffffffffa0dc4a80>] ? osp_sync_thread+0x240/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffffa0dc4840>] ? osp_sync_thread+0x0/0x7e0 [osp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
[root@puma-mds-10-6 etc]#