Aug 5 11:39:08 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1470422342/real 0] req@ffff880172ee4080 x1541832703172272/t0(0) o38->soaked-MDT0000-lwp-OST0017@192.168.1.108@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1470422348 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 5 11:39:08 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#2 stuck for 67s! [ll_ost_io00_041:30832] Aug 5 11:39:21 lola-7 kernel: Modules linked in: Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#4 stuck for 67s! [ll_ost_io01_006:24627] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 4 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 24627, comm: ll_ost_io01_006 Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __read_lock_failed+0x8/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8807eb2e39d8 EFLAGS: 00000297 Aug 5 11:39:21 lola-7 kernel: RAX: ffff8806ce315db0 RBX: ffff8807eb2e39e0 RCX: 0000000057a4dd06 Aug 5 11:39:21 lola-7 kernel: RDX: 0000000000000000 RSI: ffff8806ce3149a8 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: 0000000000000000 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 0000000000000039 R12: ffffffff8153cde6 Aug 5 11:39:21 lola-7 kernel: R13: ffff8807eb2e3978 R14: ffff88017c17e000 R15: ffff8806ce3149a8 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff880038680000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 00007fa0f09d9000 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io01_006 (pid: 24627, threadinfo ffff8807eb2e0000, task ffff88081acd6ab0) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153ce94 ffff8807eb2e3a20 ffffffffa0fe990a ffff8807eb2e3a20 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 5 11:39:21 lola-7 kernel: ffff8803dd948c40 ffff8807eb2e3ab0 ffffffffa112a746 00000001013488db Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _read_lock+0x14/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_attr_get+0xaa/0x2e0 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0xb16/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 90 83 3f 01 <78> f9 f0 ff 0f 0f 88 ed ff ff ff c3 90 90 90 90 90 90 90 90 90 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _read_lock+0x14/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_attr_get+0xaa/0x2e0 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0xb16/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#9 stuck for 67s! [ll_ost_io02_018:30476] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 9 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 30476, comm: ll_ost_io02_018 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff880357143998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8803571439a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 00000039ffb00000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880616828b00 R09: 00000000ffffffff Aug 5 11:39:21 lola-7 kernel: R10: 00000000ffffffff R11: 00000000ffffffff R12: ffff8806b34d2648 Aug 5 11:39:21 lola-7 kernel: R13: ffff8806b34d26c0 R14: ffff8806b34d26a0 R15: ffff8806b34d2648 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e420000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 0000000001ba93a8 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io02_018 (pid: 30476, threadinfo ffff880357140000, task ffff880361b50ab0) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff880357143a20 ffffffffa0fea5fb ffff88060d942000 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff8803571439f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff880357143a20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#10 stuck for 67s! [ll_ost_io02_030:30892] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 10 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 30892, comm: ll_ost_io02_030 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff88066871f998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff88066871f9a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 00000039ffc00000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880616828b00 R09: 00000000ffffffff Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 00000000ffffffff R12: ffff8806c3c5d188 Aug 5 11:39:21 lola-7 kernel: R13: ffff8806c3c5d200 R14: ffff8806c3c5d1e0 R15: ffff8806c3c5d188 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e440000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 00007f03e110c580 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io02_030 (pid: 30892, threadinfo ffff88066871c000, task ffff88081ae4d520) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff88066871fa20 ffffffffa0fea5fb ffff880639f5a800 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff88066871f9f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff88066871fa20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#11 stuck for 67s! [ll_ost_io02_020:30793] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 11 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 30793, comm: ll_ost_io02_020 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8801c5e93938 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000000000 RBX: ffff8801c5e93940 RCX: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: RDX: ffff880038635a00 RSI: ffff8807dec08600 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: 0000000000000000 R09: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000000 R11: 00000000ffffffff R12: ffff8806cdeda5a0 Aug 5 11:39:21 lola-7 kernel: R13: ffff8806cdeda59c R14: 0000000000000000 R15: 0000000000000286 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e460000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 00007f9748ee5ec8 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io02_020 (pid: 30793, threadinfo ffff8801c5e90000, task ffff8801dab62040) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff8801c5e93970 ffffffffa0fe3ab0 ffff8807dec08600 Aug 5 11:39:21 lola-7 kernel: ffff8807dec08600 ffff8806ce3149a8 ffff8807dec08600 ffff8801c5e939a0 Aug 5 11:39:21 lola-7 kernel: ffffffffa0fe3cba 00000000000000e0 ffff880696302800 0000000000000100 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_object_sa_update+0x4a/0xd0 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x466/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#13 stuck for 67s! [ll_ost_io03_054:30876] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 13 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 30876, comm: ll_ost_io03_054 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#14 stuck for 67s! [ll_ost_io03_058:30882] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 14 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#15 stuck for 67s! [ll_ost_io03_020:30433] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 30433, comm: ll_ost_io03_020 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8803b90df998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8803b90df9a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 00000005aae00000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff88080ba37640 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 00000000ffffffff R12: ffff88063ab39648 Aug 5 11:39:21 lola-7 kernel: R13: ffff88063ab396c0 R14: ffff88063ab396a0 R15: ffff88063ab39648 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e4e0000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 00007f97473640a0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io03_020 (pid: 30433, threadinfo ffff8803b90dc000, task ffff8803b9097520) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff8803b90dfa20 ffffffffa0fea5fb ffff880616692600 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff8803b90df9f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff8803b90dfa20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#16 stuck for 67s! [ll_ost_io00_011:24611] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 16 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 24611, comm: ll_ost_io00_011 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8803eacd7998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8803eacd79a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 000000186ee00000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff8801625d6b40 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 00000000ffffffff R12: ffff8801e8908608 Aug 5 11:39:21 lola-7 kernel: R13: ffff8801e8908680 R14: ffff8801e8908660 R15: ffff8801e8908608 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff880038700000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 00007f67bb7695b6 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io00_011 (pid: 24611, threadinfo ffff8803eacd4000, task ffff8803d521aab0) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff8803eacd7a20 ffffffffa0fea5fb ffff88018397ea00 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff8803eacd79f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff8803eacd7a20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#17 stuck for 67s! [ll_ost_io00_007:24603] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Pid: 24603, comm: ll_ost_io00_007 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8807f6327998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8807f63279a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 000000186ec00000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff8801625d6b40 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 000000000000002a R12: ffff8801ec4b8ac8 Aug 5 11:39:21 lola-7 kernel: R13: ffff8801ec4b8b40 R14: ffff8801ec4b8b20 R15: ffff8801ec4b8ac8 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff880038720000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff8807f6327a20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 11:39:21 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#20 stuck for 67s! [ll_ost_io01_019:30456] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 20 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 30456, comm: ll_ost_io01_019 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8803af8f3998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff8803af8f3a20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#22 stuck for 67s! [ll_ost_io01_018:30455] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 22 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8803ac7a39a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880176dbe100 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 000000000000000d R12: ffff8803042cba88 Aug 5 11:39:21 lola-7 kernel: R13: ffff8803042cbb00 R14: ffff8803042cbae0 R15: ffff8803042cba88 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff8800387c0000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 0000000000b2d874 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff8803ac7a3a20 ffffffffa0fea5fb ffff880169f5fc00 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff8803ac7a3a20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 11:39:21 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 11:39:21 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#23 stuck for 67s! [ll_ost_io01_020:30459] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8803a52039a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: CR2: 00007fce9577dd90 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io01_020 (pid: 30459, threadinfo ffff8803a5200000, task ffff8803a51ff520) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff8803a5203a20 ffffffffa0fea5fb ffff88030fdde800 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff8803a52039f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff8803a5203a20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 11:39:21 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#24 stuck for 67s! [ll_ost_io02_021:30794] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Pid: 30794, comm: ll_ost_io02_021 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8801c50d3998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8801c50d39a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 0000000f3b600000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880616828e00 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 00000000ffffffff R12: ffff8806c0be8ac8 Aug 5 11:39:21 lola-7 kernel: R13: ffff8806c0be8b40 R14: ffff8806c0be8b20 R15: ffff8806c0be8ac8 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e500000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 00007ffd19995f90 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io02_021 (pid: 30794, threadinfo ffff8801c50d0000, task ffff8801c5075520) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff8801c50d3a20 ffffffffa0fea5fb ffff8806c89f2400 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff8801c50d39f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#25 stuck for 67s! [ll_ost_io02_034:31105] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 31105, comm: ll_ost_io02_034 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff88062b0b7998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff88062b0b79a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 0000000f3b800000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880616828e00 R09: 00000000ffffffff Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 00000000ffffffff R12: ffff8806274b8a88 Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff88062b0b79f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff88062b0b7a20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_032:31048] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 27 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 31048, comm: ll_ost_io02_032 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x7/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8806e99bf998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8806e99bf9a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 0000000f3b700000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880616828e00 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 00000000ffffffff R12: ffff88075224ebe8 Aug 5 11:39:21 lola-7 kernel: R13: ffff88075224ec60 R14: ffff88075224ec40 R15: ffff88075224ebe8 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 000000393f0700e0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: d8 3f 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 90 81 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#28 stuck for 67s! [ll_ost_io03_003:24552] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 28 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 24552, comm: ll_ost_io03_003 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8803d33d7998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8803d33d79a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 00000005aad00000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff88080ba37640 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 0000000000000001 R12: ffff8806b6853728 Aug 5 11:39:21 lola-7 kernel: R13: ffff8806b68537a0 R14: ffff8806b6853780 R15: ffff8806b6853728 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e580000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 0000003940c13010 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io03_003 (pid: 24552, threadinfo ffff8803d33d4000, task ffff8803e974d520) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff8803d33d7a20 ffffffffa0fea5fb ffff88063c4e1a00 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff8803d33d79f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff8803d33d7a20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#29 stuck for 67s! [ll_ost_io03_012:30425] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 29 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 30425, comm: ll_ost_io03_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff8803bc5af998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8803bc5af9a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 00000021f7900000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880738e9fa40 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 00000000ffffffff R12: ffff8806bd631568 Aug 5 11:39:21 lola-7 kernel: R13: ffff8806bd6315e0 R14: ffff8806bd6315c0 R15: ffff8806bd631568 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e5a0000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 0000003940c05167 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io03_012 (pid: 30425, threadinfo ffff8803bc5ac000, task ffff8803bcfe2ab0) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff8803bc5afa20 ffffffffa0fea5fb ffff880618925600 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff8803bc5af9f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff8803bc5afa20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:21 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:21 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: BUG: soft lockup - CPU#30 stuck for 67s! [ll_ost_io03_060:30886] Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 30 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 30886, comm: ll_ost_io03_060 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff88067ca7f998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff88067ca7f9a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 00000005aab00000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff88080ba37640 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 0000000000000001 R12: ffff8806d082d0a8 Aug 5 11:39:21 lola-7 kernel: R13: ffff8806d082d120 R14: ffff8806d082d100 R15: ffff8806d082d0a8 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e5c0000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 000000393f0acd90 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io03_060 (pid: 30886, threadinfo ffff88067ca7c000, task ffff88068d23b520) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff88067ca7fa20 ffffffffa0fea5fb ffff880618925800 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff88067ca7f9f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff88067ca7fa20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: LNetError: 17084:0:(o2iblnd_cb.c:3114:kiblnd_check_txs_locked()) Timed out tx: active_txs, 3 seconds Aug 5 11:39:21 lola-7 kernel: LNetError: 17084:0:(o2iblnd_cb.c:3177:kiblnd_check_conns()) Timed out RDMA with 192.168.1.111@o2ib10 (74): c: 0, oc: 0, rc: 8 Aug 5 11:39:21 lola-7 kernel: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: CPU 2 Aug 5 11:39:21 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:21 lola-7 kernel: Aug 5 11:39:21 lola-7 kernel: Pid: 30832, comm: ll_ost_io00_041 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:21 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:39:21 lola-7 kernel: RSP: 0018:ffff88067df83998 EFLAGS: 00000287 Aug 5 11:39:21 lola-7 kernel: RAX: 0000000000001000 RBX: ffff88067df839a0 RCX: ffff8803e81e112c Aug 5 11:39:21 lola-7 kernel: RDX: 000000186f000000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:21 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff8801625d6b40 R09: dead000000100100 Aug 5 11:39:21 lola-7 kernel: R10: 0000000000000001 R11: 0000000000000047 R12: ffff880312b91a88 Aug 5 11:39:21 lola-7 kernel: R13: ffff880312b91b00 R14: ffff880312b91ae0 R15: ffff880312b91a88 Aug 5 11:39:21 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff880038640000(0000) knlGS:0000000000000000 Aug 5 11:39:21 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:21 lola-7 kernel: CR2: 00000000019fa1f0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:21 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:21 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:21 lola-7 kernel: Process ll_ost_io00_041 (pid: 30832, threadinfo ffff88067df80000, task ffff88067df7d520) Aug 5 11:39:21 lola-7 kernel: Stack: Aug 5 11:39:21 lola-7 kernel: ffffffff8153d007 ffff88067df83a20 ffffffffa0fea5fb ffff8801f9a92800 Aug 5 11:39:21 lola-7 kernel: 0000000000000000 ffff88067df839f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:21 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff88067df83a20 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:21 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:39:21 lola-7 kernel: Call Trace: Aug 5 11:39:21 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:21 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:21 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:21 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:39:21 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:21 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:21 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:21 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:24 lola-7 kernel: BUG: soft lockup - CPU#0 stuck for 67s! [ll_ost_io00_029:30447] Aug 5 11:39:24 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) Aug 5 11:39:24 lola-7 kernel: BUG: soft lockup - CPU#3 stuck for 67s! [ll_ost_io00_045:30848] Aug 5 11:39:24 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:24 lola-7 kernel: CPU 3 Aug 5 11:39:24 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:24 lola-7 kernel: Aug 5 11:39:24 lola-7 kernel: Pid: 30848, comm: ll_ost_io00_045 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:24 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:39:24 lola-7 kernel: RSP: 0018:ffff8801bc6cf998 EFLAGS: 00000287 Aug 5 11:39:24 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8801bc6cf9a0 RCX: 0000000000000000 Aug 5 11:39:24 lola-7 kernel: RDX: 0000001337f30000 RSI: 0000000000000008 RDI: ffff8806ce314a60 Aug 5 11:39:24 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880173728e78 R09: 0000000000000000 Aug 5 11:39:24 lola-7 kernel: R10: 0000000000000001 R11: 0000000000000000 R12: 0000001337f2f000 Aug 5 11:39:24 lola-7 kernel: R13: 0000000000001000 R14: ffff8801bc6cf920 R15: ffffffffa037d550 Aug 5 11:39:24 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff880038660000(0000) knlGS:0000000000000000 Aug 5 11:39:24 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:24 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:24 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:24 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:24 lola-7 kernel: Process ll_ost_io00_045 (pid: 30848, threadinfo ffff8801bc6cc000, task ffff8801bc577520) Aug 5 11:39:24 lola-7 kernel: Stack: Aug 5 11:39:24 lola-7 kernel: ffffffff8153d007 ffff8801bc6cfa20 ffffffffa0fea5fb ffff8803bb492400 Aug 5 11:39:24 lola-7 kernel: 0000000000000000 ffffea000da82e68 ffff8801f9077300 ffff8803e81e0000 Aug 5 11:39:24 lola-7 kernel: ffff8803e81e112c 00000000000a3000 000000a3ce3149a8 ffff8801bc6cfa20 Aug 5 11:39:24 lola-7 kernel: Call Trace: Aug 5 11:39:24 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:24 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:24 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:24 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:24 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:24 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 11:39:24 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:24 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:39:24 lola-7 kernel: Call Trace: Aug 5 11:39:24 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:24 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:24 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:24 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:24 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:24 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 11:39:24 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:24 lola-7 kernel: BUG: soft lockup - CPU#26 stuck for 67s! [ll_ost_io02_035:31106] Aug 5 11:39:24 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:24 lola-7 kernel: CPU 26 Aug 5 11:39:24 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:24 lola-7 kernel: Aug 5 11:39:24 lola-7 kernel: Pid: 31106, comm: ll_ost_io02_035 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:24 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:39:24 lola-7 kernel: RSP: 0018:ffff88061107f998 EFLAGS: 00000287 Aug 5 11:39:24 lola-7 kernel: RAX: 0000000000001000 RBX: ffff88061107f9a0 RCX: ffff8803e81e112c Aug 5 11:39:24 lola-7 kernel: RDX: 00000039fff00000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:24 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880616828b00 R09: 0000000000000000 Aug 5 11:39:24 lola-7 kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffff8806e8af00a8 Aug 5 11:39:24 lola-7 kernel: R13: ffff8806e8af0120 R14: ffff8806e8af0100 R15: ffff8806e8af00a8 Aug 5 11:39:24 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e540000(0000) knlGS:0000000000000000 Aug 5 11:39:24 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:24 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:24 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:24 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:24 lola-7 kernel: Process ll_ost_io02_035 (pid: 31106, threadinfo ffff88061107c000, task ffff88065531eab0) Aug 5 11:39:24 lola-7 kernel: Stack: Aug 5 11:39:24 lola-7 kernel: ffffffff8153d007 ffff88061107fa20 ffffffffa0fea5fb ffff8807b450fa00 Aug 5 11:39:24 lola-7 kernel: 0000000000000000 ffff88061107f9f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:24 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff88061107fa20 Aug 5 11:39:24 lola-7 kernel: Call Trace: Aug 5 11:39:24 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:24 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:24 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:24 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:24 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:24 lola-7 kernel: [] ? __mutex_lock_slowpath+0x1d4/0x210 Aug 5 11:39:24 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:24 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:39:24 lola-7 kernel: Call Trace: Aug 5 11:39:24 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:24 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:24 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:24 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:24 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:24 lola-7 kernel: [] ? __mutex_lock_slowpath+0x1d4/0x210 Aug 5 11:39:24 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:24 lola-7 kernel: BUG: soft lockup - CPU#31 stuck for 67s! [ll_ost_io03_045:30858] Aug 5 11:39:24 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:24 lola-7 kernel: CPU 31 Aug 5 11:39:24 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:39:24 lola-7 kernel: Aug 5 11:39:24 lola-7 kernel: Pid: 30858, comm: ll_ost_io03_045 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:39:24 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x7/0x20 Aug 5 11:39:24 lola-7 kernel: RSP: 0018:ffff8806870ab998 EFLAGS: 00000287 Aug 5 11:39:24 lola-7 kernel: RAX: 0000000000001000 RBX: ffff8806870ab9a0 RCX: ffff8803e81e112c Aug 5 11:39:24 lola-7 kernel: RDX: 00000021f7b00000 RSI: 0000000000000000 RDI: ffff8806ce314a60 Aug 5 11:39:24 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880738e9fa40 R09: dead000000100100 Aug 5 11:39:24 lola-7 kernel: R10: 0000000000000001 R11: 0000000000000035 R12: ffff88082c0ad068 Aug 5 11:39:24 lola-7 kernel: R13: ffff88082c0ad0e0 R14: ffff88082c0ad0c0 R15: ffff88082c0ad068 Aug 5 11:39:24 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e5e0000(0000) knlGS:0000000000000000 Aug 5 11:39:24 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:39:24 lola-7 kernel: CR2: 00007f97473640a0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:39:24 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:39:24 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:39:24 lola-7 kernel: Process ll_ost_io03_045 (pid: 30858, threadinfo ffff8806870a8000, task ffff8807b5e04ab0) Aug 5 11:39:24 lola-7 kernel: Stack: Aug 5 11:39:24 lola-7 kernel: ffffffff8153d007 ffff8806870aba20 ffffffffa0fea5fb ffff880626603400 Aug 5 11:39:24 lola-7 kernel: 0000000000000000 ffff8806870ab9f0 ffffffffa0fdcd19 ffff8803e81e0000 Aug 5 11:39:24 lola-7 kernel: ffff8803e81e112c 0000000000100000 00000100ce3149a8 ffff8806870aba20 Aug 5 11:39:24 lola-7 kernel: Call Trace: Aug 5 11:39:24 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:24 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:24 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:24 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:24 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:24 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:39:24 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:39:24 lola-7 kernel: Code: d8 3f 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 90 81 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff Aug 5 11:39:24 lola-7 kernel: Call Trace: Aug 5 11:39:24 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:24 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:24 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:39:24 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:39:24 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:24 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:39:24 lola-7 kernel: [] ? osd_write_commit+0x34b/0x590 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? osd_trans_start+0x139/0x430 [osd_zfs] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw_write+0x664/0xfa0 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:39:24 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:39:24 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:39:24 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:39:24 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 11:39:24 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:39:24 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:39:24 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 11:39:24 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:39:26 lola-7 kernel: Lustre: soaked-OST0017: Client 3cc8f03d-59b0-76b4-60ec-19a13d95af15 (at 192.168.1.122@o2ib100) reconnecting Aug 5 11:39:26 lola-7 kernel: Lustre: Skipped 5 previous similar messages Aug 5 11:39:26 lola-7 kernel: Lustre: soaked-OST0017: Connection restored to 3cc8f03d-59b0-76b4-60ec-19a13d95af15 (at 192.168.1.122@o2ib100) Aug 5 11:39:26 lola-7 kernel: Lustre: Skipped 5 previous similar messages Aug 5 11:39:26 lola-7 zed[10312]: Invoking "all-syslog.sh" eid=50 pid=31128 Aug 5 11:39:26 lola-7 zed: eid=50 class=delay pool=soaked-ost11 Aug 5 11:39:26 lola-7 zed[10312]: Finished "all-syslog.sh" eid=50 pid=31128 exit=0 Aug 5 11:39:26 lola-7 zed[10312]: Invoking "all-syslog.sh" eid=51 pid=31130 Aug 5 11:39:26 lola-7 kernel: Lustre: 24597:0:(service.c:1951:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 84s req@ffff880824123450 x1541009565658768/t0(0) o4->96018fae-8d83-4949-5d4f-b8e2450e2ea9@192.168.1.129@o2ib100:-1/-1 lens 608/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 Aug 5 11:39:26 lola-7 kernel: Lustre: 30879:0:(service.c:1951:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 84s req@ffff880824124c50 x1541009565658800/t0(0) o4->96018fae-8d83-4949-5d4f-b8e2450e2ea9@192.168.1.129@o2ib100:-1/-1 lens 608/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 Aug 5 11:39:26 lola-7 kernel: Lustre: 31108:0:(service.c:1951:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 28s req@ffff8807a5829c50 x1541009687170272/t0(0) o4->5041fba2-c9a3-14b2-8ec5-ce966cd27ea6@192.168.1.117@o2ib100:-1/-1 lens 608/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 Aug 5 11:39:26 lola-7 kernel: Lustre: 30879:0:(service.c:1951:ptlrpc_server_handle_req_in()) Skipped 38 previous similar messages Aug 5 11:39:26 lola-7 kernel: Lustre: 31108:0:(service.c:1951:ptlrpc_server_handle_req_in()) Skipped 38 previous similar messages Aug 5 11:39:26 lola-7 kernel: Lustre: 30458:0:(service.c:2103:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:1s); client may timeout. req@ffff880824bf6050 x1541009472616320/t0(0) o4->3cc8f03d-59b0-76b4-60ec-19a13d95af15@192.168.1.122@o2ib100:-1/-1 lens 608/0 e 0 to 0 dl 1470422365 ref 2 fl Interpret:/2/ffffffff rc 0/-1 Aug 5 11:39:26 lola-7 kernel: Lustre: ost_io: This server is not able to keep up with request traffic (cpu-bound). Aug 5 11:39:26 lola-7 kernel: Lustre: 30458:0:(service.c:2103:ptlrpc_server_handle_request()) Skipped 41 previous similar messages Aug 5 11:39:26 lola-7 kernel: Lustre: 31108:0:(service.c:1501:ptlrpc_at_check_timed()) earlyQ=38 reqQ=0 recA=42, svcEst=88, delay=5594(jiff) Aug 5 11:39:26 lola-7 kernel: Lustre: 17107:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1470422330/real 1470422366] req@ffff8801e8fbf980 x1541832703172032/t0(0) o400->soaked-MDT0000-lwp-OST0005@192.168.1.108@o2ib10:12/10 lens 224/224 e 0 to 1 dl 1470422374 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 5 11:39:26 lola-7 kernel: Lustre: 17107:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 5 11:39:26 lola-7 kernel: LustreError: 30458:0:(service.c:2065:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.1.127@o2ib100: deadline 43:1s ago Aug 5 11:39:26 lola-7 kernel: req@ffff8808249dec50 x1541009643927728/t0(0) o4->9cff6698-27dd-142a-724f-49bd7985bea9@192.168.1.127@o2ib100:-1/-1 lens 608/0 e 0 to 0 dl 1470422365 ref 2 fl Interpret:/2/ffffffff rc 0/-1 Aug 5 11:39:26 lola-7 kernel: LustreError: 30458:0:(service.c:2065:ptlrpc_server_handle_request()) Skipped 18 previous similar messages Aug 5 11:39:26 lola-7 kernel: Lustre: 17160:0:(service.c:1289:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-45s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8801e24019c0 x1541019921601584/t0(0) o103->8dfde31b-b4ea-ae0b-171e-8bd61886ac26@192.168.1.124@o2ib100:-1/-1 lens 328/0 e 0 to 0 dl 1470422321 ref 2 fl New:H/0/ffffffff rc 0/-1 Aug 5 11:39:26 lola-7 kernel: Lustre: 17160:0:(service.c:1289:ptlrpc_at_send_early_reply()) Skipped 78 previous similar messages Aug 5 11:39:26 lola-7 kernel: LustreError: 30795:0:(ldlm_lib.c:3137:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8808249e3850 x1541009499262736/t0(0) o4->9edc0a3a-fdd6-2450-a02c-64e7f43a9b80@192.168.1.130@o2ib100:-1/-1 lens 608/448 e 0 to 0 dl 1470422337 ref 1 fl Interpret:/0/0 rc 0/0 Aug 5 11:39:26 lola-7 kernel: Lustre: soaked-OST000b: Bulk IO write error with 9edc0a3a-fdd6-2450-a02c-64e7f43a9b80 (at 192.168.1.130@o2ib100), client will retry: rc -107 Aug 5 11:39:26 lola-7 kernel: Lustre: 24597:0:(service.c:1951:ptlrpc_server_handle_req_in()) Skipped 115 previous similar messages Aug 5 11:39:28 lola-7 kernel: Lustre: 31111:0:(service.c:2103:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:46s); client may timeout. req@ffff880824bf6c50 x1541009499247024/t47244714491(0) o4->9edc0a3a-fdd6-2450-a02c-64e7f43a9b80@192.168.1.130@o2ib100:-1/-1 lens 608/416 e 0 to 0 dl 1470422322 ref 1 fl Complete:/0/0 rc 0/0 Aug 5 11:39:28 lola-7 kernel: Lustre: 31111:0:(service.c:2103:ptlrpc_server_handle_request()) Skipped 148 previous similar messages Aug 5 11:39:36 lola-7 kernel: Lustre: soaked-OST0005: Client 96018fae-8d83-4949-5d4f-b8e2450e2ea9 (at 192.168.1.129@o2ib100) reconnecting Aug 5 11:39:36 lola-7 kernel: Lustre: Skipped 38 previous similar messages Aug 5 11:39:37 lola-7 kernel: Lustre: 17179:0:(service.c:2103:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:75s); client may timeout. req@ffff8808252efc50 x1541788315368208/t0(0) o5->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 432/400 e 0 to 0 dl 1470422302 ref 1 fl Complete:/0/0 rc 0/0 Aug 5 11:39:38 lola-7 kernel: Lustre: soaked-OST0017: deleting orphan objects from 0x0:2429970 to 0x0:2432657 Aug 5 11:39:40 lola-7 kernel: Lustre: 30493:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1470422373/real 1470422373] req@ffff8806d5d67380 x1541832703172288/t0(0) o104->soaked-OST0017@192.168.1.124@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1470422380 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Aug 5 11:39:40 lola-7 kernel: Lustre: 30493:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 5 11:39:44 lola-7 kernel: LustreError: 17179:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) ### client (nid 192.168.1.124@o2ib100) failed to reply to blocking AST (req@ffff880740a57980 x1541832703172576 status 0 rc -11), evict it ns: filter-soaked-OST0017_UUID lock: ffff88064ee31dc0/0xa6ec3ff74a87fb lrc: 4/0,0 mode: PR/PR res: [0x24b858:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.124@o2ib100 remote: 0x4c35d1dae8d035c1 expref: 1369 pid: 30503 timeout: 4315438495 lvb_type: 1 Aug 5 11:39:44 lola-7 kernel: LustreError: 138-a: soaked-OST0017: A client on nid 192.168.1.124@o2ib100 was evicted due to a lock blocking callback time out: rc -11 Aug 5 11:39:45 lola-7 kernel: Lustre: 30493:0:(service.c:2103:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43:64s); client may timeout. req@ffff8807caf32850 x1541788319141504/t47244729660(0) o6->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 664/400 e 0 to 0 dl 1470422321 ref 1 fl Complete:/0/0 rc 0/0 Aug 5 11:39:48 lola-7 kernel: Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9210 to 0x740000401:9857 Aug 5 11:39:48 lola-7 kernel: Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:8953 to 0x440000400:10305 Aug 5 11:39:48 lola-7 kernel: Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:8956 to 0x8c0000400:9537 Aug 5 11:39:48 lola-7 kernel: Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9082 to 0x5c0000401:9409 Aug 5 11:39:57 lola-7 kernel: Lustre: 30417:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1470422390/real 1470422390] req@ffff8806a853d980 x1541832703172960/t0(0) o104->soaked-OST0017@192.168.1.119@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1470422397 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 5 11:39:57 lola-7 kernel: Lustre: 30417:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Aug 5 11:40:01 lola-7 CROND[31157]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Aug 5 11:40:01 lola-7 CROND[31158]: (root) CMD (/usr/lib64/sa/sa1 1 1) Aug 5 11:40:03 lola-7 kernel: Lustre: soaked-OST0005: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Aug 5 11:40:03 lola-7 kernel: Lustre: soaked-OST000b: Connection restored to 192.168.1.108@o2ib10 (at 192.168.1.108@o2ib10) Aug 5 11:40:03 lola-7 kernel: Lustre: soaked-OST0011: Connection restored to 192.168.1.108@o2ib10 (at 192.168.1.108@o2ib10) Aug 5 11:40:03 lola-7 kernel: Lustre: Skipped 73 previous similar messages Aug 5 11:40:03 lola-7 kernel: Lustre: Skipped 73 previous similar messages Aug 5 11:40:03 lola-7 kernel: Lustre: Skipped 24 previous similar messages Aug 5 11:40:03 lola-7 kernel: Lustre: soaked-OST0011: deleting orphan objects from 0x0:2444397 to 0x0:2449745 Aug 5 11:40:03 lola-7 kernel: Lustre: soaked-OST000b: deleting orphan objects from 0x0:2442120 to 0x0:2446689 Aug 5 11:40:03 lola-7 kernel: Lustre: soaked-OST0005: deleting orphan objects from 0x0:2421458 to 0x0:2429649 Aug 5 11:40:04 lola-7 kernel: LustreError: 30515:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) ### client (nid 192.168.1.119@o2ib100) failed to reply to blocking AST (req@ffff88068830f0c0 x1541832703173280 status 0 rc -11), evict it ns: filter-soaked-OST0017_UUID lock: ffff88065d646d00/0xa6ec3ff74a747c lrc: 4/0,0 mode: PR/PR res: [0x24b852:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.119@o2ib100 remote: 0xba8a3408f8f47d2c expref: 1080 pid: 30419 timeout: 4315458709 lvb_type: 1 Aug 5 11:40:04 lola-7 kernel: LustreError: 138-a: soaked-OST0017: A client on nid 192.168.1.119@o2ib100 was evicted due to a lock blocking callback time out: rc -11 Aug 5 11:40:04 lola-7 kernel: LustreError: Skipped 1 previous similar message Aug 5 11:40:04 lola-7 kernel: LustreError: 30489:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8807ca51ccc0 x1541832703172784/t0(0) o104->soaked-OST0017@192.168.1.119@o2ib100:15/16 lens 296/224 e 0 to 0 dl 1470422406 ref 1 fl Rpc:ES/2/ffffffff rc -5/-1 Aug 5 11:40:04 lola-7 kernel: LustreError: 17160:0:(ldlm_lockd.c:2370:ldlm_cancel_handler()) ldlm_cancel from 192.168.1.119@o2ib100 arrived at 1470422404 with bad export cookie 46984605608730822 Aug 5 11:40:04 lola-7 kernel: LustreError: 17160:0:(ldlm_lockd.c:2370:ldlm_cancel_handler()) Skipped 2 previous similar messages Aug 5 11:40:04 lola-7 kernel: LustreError: 30515:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) Skipped 6 previous similar messages Aug 5 11:40:06 lola-7 kernel: LustreError: 30515:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88068830f0c0 x1541832703175632/t0(0) o104->soaked-OST0017@192.168.1.119@o2ib100:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Aug 5 11:40:06 lola-7 kernel: LustreError: 30515:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) ### client (nid 192.168.1.119@o2ib100) failed to reply to blocking AST (req@ffff88068830f0c0 x1541832703175632 status 0 rc -5), evict it ns: filter-soaked-OST0017_UUID lock: ffff88064e245940/0xa6ec3ff74a7f9d lrc: 4/0,0 mode: PR/PR res: [0x24b87d:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.119@o2ib100 remote: 0xba8a3408f8f47fc5 expref: 1061 pid: 24623 timeout: 4315465651 lvb_type: 1 Aug 5 11:40:06 lola-7 kernel: LustreError: 138-a: soaked-OST0017: A client on nid 192.168.1.119@o2ib100 was evicted due to a lock blocking callback time out: rc -5 Aug 5 11:40:06 lola-7 kernel: LustreError: Skipped 5 previous similar messages Aug 5 11:40:06 lola-7 kernel: LustreError: 30515:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) Skipped 1 previous similar message Aug 5 11:40:09 lola-7 kernel: LustreError: 30481:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88063ee816c0 x1541832703176480/t0(0) o104->soaked-OST0017@192.168.1.119@o2ib100:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Aug 5 11:40:09 lola-7 kernel: LustreError: 30481:0:(client.c:1168:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Aug 5 11:40:09 lola-7 kernel: LustreError: 30481:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) ### client (nid 192.168.1.119@o2ib100) failed to reply to blocking AST (req@ffff88063ee816c0 x1541832703176480 status 0 rc -5), evict it ns: filter-soaked-OST0017_UUID lock: ffff880649c59d40/0xa6ec3ff74a885d lrc: 4/0,0 mode: PR/PR res: [0x24b89e:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.119@o2ib100 remote: 0xba8a3408f8f481ee expref: 1039 pid: 17239 timeout: 4315468617 lvb_type: 1 Aug 5 11:40:09 lola-7 kernel: LustreError: 138-a: soaked-OST0017: A client on nid 192.168.1.119@o2ib100 was evicted due to a lock blocking callback time out: rc -5 Aug 5 11:40:09 lola-7 kernel: LustreError: Skipped 2 previous similar messages Aug 5 11:40:09 lola-7 kernel: LustreError: 30481:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) Skipped 2 previous similar messages Aug 5 11:40:09 lola-7 kernel: LustreError: 30511:0:(ldlm_lockd.c:2370:ldlm_cancel_handler()) ldlm_cancel from 192.168.1.124@o2ib100 arrived at 1470422409 with bad export cookie 46984605608730402 Aug 5 11:40:09 lola-7 kernel: LustreError: 30502:0:(ldlm_lockd.c:2370:ldlm_cancel_handler()) ldlm_cancel from 192.168.1.124@o2ib100 arrived at 1470422409 with bad export cookie 46984605608730402 Aug 5 11:40:09 lola-7 kernel: LustreError: 30502:0:(ldlm_lockd.c:2370:ldlm_cancel_handler()) Skipped 5 previous similar messages Aug 5 11:40:09 lola-7 kernel: LustreError: 30511:0:(ldlm_lockd.c:2370:ldlm_cancel_handler()) Skipped 2 previous similar messages Aug 5 11:40:11 lola-7 kernel: LustreError: 30485:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8806a7a16980 x1541832703177200/t0(0) o104->soaked-OST0017@192.168.1.119@o2ib100:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Aug 5 11:40:11 lola-7 kernel: LustreError: 30485:0:(client.c:1168:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Aug 5 11:40:18 lola-7 kernel: LustreError: 30482:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8806107dd3c0 x1541832703177696/t0(0) o104->soaked-OST0017@192.168.1.119@o2ib100:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Aug 5 11:40:18 lola-7 kernel: LustreError: 30482:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) ### client (nid 192.168.1.119@o2ib100) failed to reply to blocking AST (req@ffff8806107dd3c0 x1541832703177696 status 0 rc -5), evict it ns: filter-soaked-OST0017_UUID lock: ffff88065dcc76c0/0xa6ec3ff74abeb2 lrc: 4/0,0 mode: PR/PR res: [0x24b8b4:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.119@o2ib100 remote: 0xba8a3408f8f49026 expref: 986 pid: 17239 timeout: 4315477151 lvb_type: 1 Aug 5 11:40:18 lola-7 kernel: LustreError: 30482:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) Skipped 6 previous similar messages Aug 5 11:40:18 lola-7 kernel: LustreError: 138-a: soaked-OST0017: A client on nid 192.168.1.119@o2ib100 was evicted due to a lock blocking callback time out: rc -5 Aug 5 11:40:18 lola-7 kernel: LustreError: Skipped 7 previous similar messages Aug 5 11:40:30 lola-7 kernel: Lustre: 30489:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1470422423/real 1470422423] req@ffff88063ee81cc0 x1541832703176688/t0(0) o104->soaked-OST0011@192.168.1.121@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1470422430 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 5 11:40:30 lola-7 kernel: Lustre: 30489:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 26 previous similar messages Aug 5 11:40:39 lola-7 kernel: Lustre: soaked-OST0011: Client c0d67e29-a13e-9b98-1a31-172a2686134a (at 192.168.1.121@o2ib100) reconnecting Aug 5 11:40:39 lola-7 kernel: Lustre: Skipped 3 previous similar messages Aug 5 11:40:40 lola-7 kernel: LustreError: 17239:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) ### client (nid 192.168.1.121@o2ib100) failed to reply to blocking AST (req@ffff8806107dd9c0 x1541832703176928 status 0 rc -11), evict it ns: filter-soaked-OST0011_UUID lock: ffff8806c02614c0/0xa6ec3ff759928e lrc: 4/0,0 mode: PR/PR res: [0x2542f3:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.121@o2ib100 remote: 0x88bcbaa350bce12d expref: 201 pid: 24608 timeout: 4315447001 lvb_type: 1 Aug 5 11:40:40 lola-7 kernel: LustreError: 138-a: soaked-OST0011: A client on nid 192.168.1.121@o2ib100 was evicted due to a lock blocking callback time out: rc -11 Aug 5 11:40:40 lola-7 kernel: LustreError: 30511:0:(ldlm_lockd.c:2370:ldlm_cancel_handler()) ldlm_cancel from 192.168.1.121@o2ib100 arrived at 1470422439 with bad export cookie 46984605608730430 Aug 5 11:40:40 lola-7 kernel: LustreError: 30511:0:(ldlm_lockd.c:2370:ldlm_cancel_handler()) Skipped 1 previous similar message Aug 5 11:40:40 lola-7 kernel: LustreError: 17239:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) Skipped 7 previous similar messages Aug 5 11:40:40 lola-7 kernel: LustreError: 24608:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88060bdd0cc0 x1541832703177920/t0(0) o104->soaked-OST0011@192.168.1.121@o2ib100:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Aug 5 11:40:52 lola-7 TIME: Time stamp for console Aug 5 11:40:52 lola-7 zed: eid=51 class=delay pool=soaked-ost23 Aug 5 11:40:52 lola-7 zed[10312]: Finished "all-syslog.sh" eid=51 pid=31130 exit=0 Aug 5 11:40:52 lola-7 zed[10312]: Invoking "all-syslog.sh" eid=52 pid=31164 Aug 5 11:40:52 lola-7 zed: eid=52 class=delay pool=soaked-ost11 Aug 5 11:40:52 lola-7 zed[10312]: Finished "all-syslog.sh" eid=52 pid=31164 exit=0 Aug 5 11:40:52 lola-7 zed[10312]: Invoking "all-syslog.sh" eid=53 pid=31166 Aug 5 11:40:52 lola-7 zed: eid=53 class=delay pool=soaked-ost11 Aug 5 11:40:52 lola-7 zed[10312]: Finished "all-syslog.sh" eid=53 pid=31166 exit=0 Aug 5 11:40:52 lola-7 zed[10312]: Invoking "all-syslog.sh" eid=54 pid=31168 Aug 5 11:40:52 lola-7 zed: eid=54 class=delay pool=soaked-ost23 Aug 5 11:40:52 lola-7 zed[10312]: Finished "all-syslog.sh" eid=54 pid=31168 exit=0 Aug 5 11:40:52 lola-7 zed[10312]: Invoking "all-syslog.sh" eid=55 pid=31170 Aug 5 11:40:52 lola-7 zed: eid=55 class=delay pool=soaked-ost23 Aug 5 11:40:52 lola-7 zed[10312]: Finished "all-syslog.sh" eid=55 pid=31170 exit=0 Aug 5 11:40:52 lola-7 zed[10312]: Invoking "all-syslog.sh" eid=56 pid=31172 Aug 5 11:40:52 lola-7 zed: eid=56 class=delay pool=soaked-ost23 Aug 5 11:40:52 lola-7 zed[10312]: Finished "all-syslog.sh" eid=56 pid=31172 exit=0 Aug 5 11:40:52 lola-7 zed[10312]: Invoking "all-syslog.sh" eid=57 pid=31174 Aug 5 11:40:52 lola-7 zed: eid=57 class=delay pool=soaked-ost23 Aug 5 11:40:52 lola-7 zed[10312]: Finished "all-syslog.sh" eid=57 pid=31174 exit=0 Aug 5 11:40:52 lola-7 zed[10312]: Invoking "all-syslog.sh" eid=58 pid=31176 Aug 5 11:40:52 lola-7 zed: eid=58 class=delay pool=soaked-ost23 Aug 5 11:40:52 lola-7 zed[10312]: Finished "all-syslog.sh" eid=58 pid=31176 exit=0 Aug 5 11:41:15 lola-7 kernel: LustreError: 30515:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88060d4269c0 x1541832703279248/t0(0) o104->soaked-OST0017@192.168.1.119@o2ib100:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Aug 5 11:41:15 lola-7 kernel: LustreError: 30515:0:(client.c:1168:ptlrpc_import_delay_req()) Skipped 12 previous similar messages Aug 5 11:41:15 lola-7 kernel: LustreError: 30515:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) ### client (nid 192.168.1.119@o2ib100) failed to reply to blocking AST (req@ffff88060d4269c0 x1541832703279248 status 0 rc -5), evict it ns: filter-soaked-OST0017_UUID lock: ffff88065d21a5c0/0xa6ec3ff74a5cf8 lrc: 4/0,0 mode: PR/PR res: [0x24b8b5:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.119@o2ib100 remote: 0xba8a3408f8f472cf expref: 629 pid: 17180 timeout: 4315534595 lvb_type: 1 Aug 5 11:41:15 lola-7 kernel: LustreError: 30515:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) Skipped 9 previous similar messages Aug 5 11:41:15 lola-7 kernel: LustreError: 138-a: soaked-OST0017: A client on nid 192.168.1.119@o2ib100 was evicted due to a lock blocking callback time out: rc -5 Aug 5 11:41:15 lola-7 kernel: LustreError: Skipped 20 previous similar messages Aug 5 11:41:29 lola-7 kernel: Lustre: soaked-OST0011: Connection restored to c0d67e29-a13e-9b98-1a31-172a2686134a (at 192.168.1.121@o2ib100) Aug 5 11:41:29 lola-7 kernel: Lustre: Skipped 7 previous similar messages Aug 5 11:41:48 lola-7 kernel: LustreError: 30515:0:(client.c:1168:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88062109c980 x1541832703281408/t0(0) o104->soaked-OST0017@192.168.1.119@o2ib100:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Aug 5 11:41:48 lola-7 kernel: LustreError: 30515:0:(client.c:1168:ptlrpc_import_delay_req()) Skipped 7 previous similar messages Aug 5 11:41:48 lola-7 kernel: LustreError: 30515:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) ### client (nid 192.168.1.119@o2ib100) failed to reply to blocking AST (req@ffff88062109c980 x1541832703281408 status 0 rc -5), evict it ns: filter-soaked-OST0017_UUID lock: ffff8806521bd780/0xa6ec3ff74a6c41 lrc: 4/0,0 mode: PR/PR res: [0x24b92c:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.119@o2ib100 remote: 0xba8a3408f8f47b57 expref: 315 pid: 17178 timeout: 4315567636 lvb_type: 1 Aug 5 11:41:48 lola-7 kernel: LustreError: 30515:0:(ldlm_lockd.c:691:ldlm_handle_ast_error()) Skipped 7 previous similar messages Aug 5 11:41:48 lola-7 kernel: LustreError: 138-a: soaked-OST0017: A client on nid 192.168.1.119@o2ib100 was evicted due to a lock blocking callback time out: rc -5 Aug 5 11:41:48 lola-7 kernel: LustreError: Skipped 7 previous similar messages Aug 5 11:43:34 lola-7 kernel: LNet: Service thread pid 30504 was inactive for 236.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 5 11:43:34 lola-7 kernel: Pid: 30504, comm: ll_ost03_036 Aug 5 11:43:34 lola-7 kernel: Aug 5 11:43:34 lola-7 kernel: Call Trace: Aug 5 11:43:34 lola-7 kernel: [] ? vdev_mirror_child_done+0x0/0x30 [zfs] Aug 5 11:43:34 lola-7 kernel: [] io_schedule+0x73/0xc0 Aug 5 11:43:34 lola-7 kernel: [] cv_wait_common+0xb1/0x130 [spl] Aug 5 11:43:34 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:43:34 lola-7 kernel: [] __cv_wait_io+0x18/0x20 [spl] Aug 5 11:43:34 lola-7 kernel: [] zio_wait+0x10b/0x1e0 [zfs] Aug 5 11:43:34 lola-7 kernel: [] dbuf_read+0x439/0x850 [zfs] Aug 5 11:43:34 lola-7 kernel: [] dmu_buf_hold+0x68/0x90 [zfs] Aug 5 11:43:34 lola-7 kernel: [] zap_get_leaf_byblk+0x85/0x2d0 [zfs] Aug 5 11:43:34 lola-7 kernel: [] zap_deref_leaf+0x71/0x90 [zfs] Aug 5 11:43:34 lola-7 kernel: [] fzap_lookup+0x60/0x120 [zfs] Aug 5 11:43:34 lola-7 kernel: [] zap_lookup_norm+0xe9/0x190 [zfs] Aug 5 11:43:34 lola-7 kernel: [] zap_lookup+0x33/0x40 [zfs] Aug 5 11:43:34 lola-7 kernel: [] dmu_tx_hold_zap+0x146/0x210 [zfs] Aug 5 11:43:34 lola-7 kernel: [] osd_declare_object_destroy+0x1f0/0x4d0 [osd_zfs] Aug 5 11:43:34 lola-7 kernel: [] ofd_object_destroy+0x244/0x8e0 [ofd] Aug 5 11:43:34 lola-7 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Aug 5 11:43:34 lola-7 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Aug 5 11:43:34 lola-7 kernel: [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 5 11:43:34 lola-7 kernel: [] ofd_create_hdl+0x1973/0x2530 [ofd] Aug 5 11:43:34 lola-7 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 5 11:43:34 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:43:34 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:43:34 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:43:34 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:43:34 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:43:34 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:43:34 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:43:34 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:43:34 lola-7 kernel: Aug 5 11:43:34 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470422614.30504 Aug 5 11:43:36 lola-7 kernel: LNet: Service thread pid 30504 completed after 238.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Aug 5 11:44:59 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:44:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:44:59 lola-7 kernel: CPU 27 Aug 5 11:44:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:44:59 lola-7 kernel: Aug 5 11:44:59 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:44:59 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:44:59 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:44:59 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:44:59 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:44:59 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:44:59 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:44:59 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:44:59 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:44:59 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:44:59 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:44:59 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:44:59 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:44:59 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:44:59 lola-7 kernel: Stack: Aug 5 11:44:59 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:44:59 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:44:59 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:44:59 lola-7 kernel: Call Trace: Aug 5 11:44:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:44:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:44:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:44:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:44:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:44:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:44:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:44:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:44:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:44:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:44:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:44:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:44:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:44:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:44:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:44:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:44:59 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:44:59 lola-7 kernel: Call Trace: Aug 5 11:44:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:44:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:44:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:44:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:44:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:44:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:44:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:44:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:44:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:44:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:44:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:44:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:44:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:44:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:44:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:44:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:44:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:44:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:46:23 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:46:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:46:23 lola-7 kernel: CPU 27 Aug 5 11:46:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:46:23 lola-7 kernel: Aug 5 11:46:23 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:46:23 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:46:23 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:46:23 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:46:23 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:46:23 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:46:23 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:46:23 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:46:23 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:46:23 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:46:23 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:46:23 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:46:23 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:46:23 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:46:23 lola-7 kernel: Stack: Aug 5 11:46:23 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:46:23 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:46:23 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:46:23 lola-7 kernel: Call Trace: Aug 5 11:46:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:46:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:46:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:46:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:46:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:46:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:46:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:46:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:46:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:46:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:46:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:46:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:46:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:46:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:46:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:46:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:46:23 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:46:23 lola-7 kernel: Call Trace: Aug 5 11:46:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:46:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:46:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:46:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:46:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:46:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:46:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:46:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:46:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:46:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:46:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:46:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:46:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:46:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:46:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:46:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:46:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:46:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:46:56 lola-7 kernel: LNet: Service thread pid 30445 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 5 11:46:56 lola-7 kernel: Pid: 30445, comm: ll_ost_io02_011 Aug 5 11:46:56 lola-7 kernel: Aug 5 11:46:56 lola-7 kernel: Call Trace: Aug 5 11:46:56 lola-7 kernel: [] ? dmu_write+0x140/0x1a0 [zfs] Aug 5 11:46:56 lola-7 kernel: [] ? lprocfs_oh_tally_log2+0x2e/0x30 [obdclass] Aug 5 11:46:56 lola-7 kernel: [] ? record_end_io+0xa2/0xe0 [osd_zfs] Aug 5 11:46:56 lola-7 kernel: [] rwsem_down_failed_common+0x95/0x1d0 Aug 5 11:46:56 lola-7 kernel: [] rwsem_down_read_failed+0x26/0x30 Aug 5 11:46:56 lola-7 kernel: [] call_rwsem_down_read_failed+0x14/0x30 Aug 5 11:46:56 lola-7 kernel: [] ? down_read+0x24/0x30 Aug 5 11:46:56 lola-7 kernel: [] osd_object_sa_dirty_rele+0x88/0x110 [osd_zfs] Aug 5 11:46:56 lola-7 kernel: [] osd_trans_stop+0x397/0x520 [osd_zfs] Aug 5 11:46:56 lola-7 kernel: [] ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:46:56 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:46:56 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:46:56 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:46:56 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:46:56 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:46:56 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:46:56 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:46:56 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:46:56 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:46:56 lola-7 kernel: Aug 5 11:46:56 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470422816.30445 Aug 5 11:46:56 lola-7 kernel: Pid: 30843, comm: ll_ost_io03_042 Aug 5 11:46:56 lola-7 kernel: Aug 5 11:46:56 lola-7 kernel: Call Trace: Aug 5 11:46:56 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:46:56 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:46:56 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:46:56 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:46:56 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:46:56 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:46:56 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:46:56 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:46:56 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:46:56 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:46:56 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:46:56 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:46:56 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:46:56 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:46:56 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:46:56 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:46:56 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:46:56 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:46:56 lola-7 kernel: Aug 5 11:46:56 lola-7 kernel: Pid: 31114, comm: ll_ost_io02_043 Aug 5 11:46:56 lola-7 kernel: Aug 5 11:46:56 lola-7 kernel: Call Trace: Aug 5 11:46:56 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 11:46:56 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:46:56 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:46:56 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:46:56 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:46:56 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:46:56 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:46:56 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:46:56 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:46:56 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:46:56 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:46:56 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:46:56 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:46:56 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:46:56 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:46:56 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:46:56 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:46:56 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:46:56 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:46:56 lola-7 kernel: Aug 5 11:46:56 lola-7 kernel: Pid: 24595, comm: ll_ost_io02_005 Aug 5 11:46:56 lola-7 kernel: Aug 5 11:46:56 lola-7 kernel: Call Trace: Aug 5 11:46:56 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 11:46:56 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:46:56 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:46:56 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:46:56 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:46:56 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:46:56 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:46:56 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:46:56 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:46:56 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:46:56 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:46:56 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:46:56 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:46:56 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:46:56 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:46:56 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:46:56 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:46:56 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:46:56 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:46:56 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:46:56 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:46:56 lola-7 kernel: Aug 5 11:46:56 lola-7 kernel: LNet: Service thread pid 30815 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 11:47:19 lola-7 kernel: LNet: Service thread pid 30428 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 11:47:19 lola-7 kernel: LNet: Skipped 31 previous similar messages Aug 5 11:47:19 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470422839.30428 Aug 5 11:47:26 lola-7 kernel: INFO: task ll_ost03_001:17179 blocked for more than 120 seconds. Aug 5 11:47:26 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:26 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:26 lola-7 kernel: ll_ost03_001 D 000000000000001e 0 17179 2 0x00000080 Aug 5 11:47:26 lola-7 kernel: ffff880824f7f940 0000000000000046 0000000000000000 0000000000000000 Aug 5 11:47:26 lola-7 kernel: ffff880824f7f8b0 ffffffffa03744f0 000012f656d0db62 0000000000000013 Aug 5 11:47:26 lola-7 kernel: ffff880824f7fa10 000000010139b104 ffff880824f665f8 ffff880824f7ffd8 Aug 5 11:47:26 lola-7 kernel: Call Trace: Aug 5 11:47:26 lola-7 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Aug 5 11:47:26 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:26 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:26 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:26 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:47:26 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:47:26 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:47:26 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:47:26 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Aug 5 11:47:26 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:26 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:26 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:26 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:26 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:26 lola-7 kernel: INFO: task ll_ost_io00_001:17190 blocked for more than 120 seconds. Aug 5 11:47:26 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:26 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:26 lola-7 kernel: ll_ost_io00_0 D 0000000000000000 0 17190 2 0x00000080 Aug 5 11:47:26 lola-7 kernel: ffff88082413f7e0 0000000000000046 000000092413f780 ffffffff0000000e Aug 5 11:47:26 lola-7 kernel: 00000011edde0258 0000000000000000 ffff88082413f7c0 0000000000000050 Aug 5 11:47:26 lola-7 kernel: 0000000000000008 0000000000000000 ffff880824133068 ffff88082413ffd8 Aug 5 11:47:26 lola-7 kernel: Call Trace: Aug 5 11:47:26 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 11:47:26 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:26 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:26 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:26 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:47:26 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:47:26 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:47:26 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:47:26 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:47:26 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:47:26 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:47:26 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:47:26 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:47:26 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:47:26 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:26 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:26 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:26 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:26 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:26 lola-7 kernel: INFO: task ll_ost_io01_001:17193 blocked for more than 120 seconds. Aug 5 11:47:26 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:26 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:26 lola-7 kernel: ll_ost_io01_0 D 0000000000000004 0 17193 2 0x00000080 Aug 5 11:47:26 lola-7 kernel: ffff8808234337e0 0000000000000046 0000000000000000 ffff8808234337a4 Aug 5 11:47:26 lola-7 kernel: 0000001100000000 ffff88043fe82600 000012f656dca30b ffff88044e575a00 Aug 5 11:47:26 lola-7 kernel: 00000000000001f9 000000010139b10d ffff880823429068 ffff880823433fd8 Aug 5 11:47:26 lola-7 kernel: Call Trace: Aug 5 11:47:26 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:26 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:26 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:26 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:47:26 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:47:26 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:47:26 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:47:26 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:47:26 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:47:26 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:47:26 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:47:26 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:47:26 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:47:26 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:26 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:26 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:26 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:26 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:26 lola-7 kernel: INFO: task ll_ost_io01_002:17194 blocked for more than 120 seconds. Aug 5 11:47:26 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:26 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:26 lola-7 kernel: ll_ost_io01_0 D 0000000000000017 0 17194 2 0x00000080 Aug 5 11:47:26 lola-7 kernel: ffff8808234377e0 0000000000000046 0000000923437780 ffffffff0000000e Aug 5 11:47:26 lola-7 kernel: 00000011edde0258 0000000000000000 ffff8808234377c0 0000000000000050 Aug 5 11:47:26 lola-7 kernel: 0000000000000008 0000000000000000 ffff8808234285f8 ffff880823437fd8 Aug 5 11:47:26 lola-7 kernel: Call Trace: Aug 5 11:47:26 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 11:47:26 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:26 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:26 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:26 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:47:26 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:47:26 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:47:26 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:47:26 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:47:26 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:47:26 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:47:26 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:47:26 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:47:26 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:47:26 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:47:26 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:26 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:26 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:26 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:26 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:26 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:26 lola-7 kernel: INFO: task ll_ost_io02_000:17195 blocked for more than 120 seconds. Aug 5 11:47:26 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:26 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:26 lola-7 kernel: ll_ost_io02_0 D 000000000000000a 0 17195 2 0x00000080 Aug 5 11:47:26 lola-7 kernel: ffff88082343f7e0 0000000000000046 ffff88082343f7a8 ffff88082343f7a4 Aug 5 11:47:26 lola-7 kernel: 0000001100000000 ffff88043fe83200 000012f662b72e8b ffff880038675a00 Aug 5 11:47:26 lola-7 kernel: 0000000000015cba 000000010139b1d4 ffff880823439ad8 ffff88082343ffd8 Aug 5 11:47:26 lola-7 kernel: Call Trace: Aug 5 11:47:26 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:26 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:27 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:47:27 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:47:27 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:47:27 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:47:27 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:47:27 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:47:27 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:47:27 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:27 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:27 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:27 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:27 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:27 lola-7 kernel: INFO: task ll_ost_io03_001:17199 blocked for more than 120 seconds. Aug 5 11:47:27 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:27 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:27 lola-7 kernel: ll_ost_io03_0 D 000000000000000f 0 17199 2 0x00000080 Aug 5 11:47:27 lola-7 kernel: ffff8808235837e0 0000000000000046 ffff8808235837a8 ffff8808235837a4 Aug 5 11:47:27 lola-7 kernel: 00000011edde0258 ffff88043fe83c00 000012f65d56614c ffff88044e535a00 Aug 5 11:47:27 lola-7 kernel: 000000000000ba1f 000000010139b177 ffff880823493068 ffff880823583fd8 Aug 5 11:47:27 lola-7 kernel: Call Trace: Aug 5 11:47:27 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:27 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:27 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:47:27 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:47:27 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:47:27 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:47:27 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 11:47:27 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:27 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:27 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:27 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:27 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:27 lola-7 kernel: INFO: task txg_quiesce:24237 blocked for more than 120 seconds. Aug 5 11:47:27 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:27 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:27 lola-7 kernel: txg_quiesce D 000000000000000a 0 24237 2 0x00000080 Aug 5 11:47:27 lola-7 kernel: ffff8803d9573d20 0000000000000046 0000000000000000 ffff8803d9573ce4 Aug 5 11:47:27 lola-7 kernel: 000012f500000000 ffff88043fe83200 000012f656aedb12 ffff88044e595a00 Aug 5 11:47:27 lola-7 kernel: 0000000000000400 000000010139b10b ffff8803d259fad8 ffff8803d9573fd8 Aug 5 11:47:27 lola-7 kernel: Call Trace: Aug 5 11:47:27 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:27 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:27 lola-7 kernel: [] ? __bitmap_weight+0x8c/0xb0 Aug 5 11:47:27 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:27 lola-7 kernel: [] txg_quiesce_thread+0x32b/0x470 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? txg_quiesce_thread+0x0/0x470 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? txg_quiesce_thread+0x0/0x470 [zfs] Aug 5 11:47:27 lola-7 kernel: [] thread_generic_wrapper+0x68/0x80 [spl] Aug 5 11:47:27 lola-7 kernel: [] ? thread_generic_wrapper+0x0/0x80 [spl] Aug 5 11:47:27 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:27 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:27 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:27 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:27 lola-7 kernel: INFO: task ll_ost_io03_003:24552 blocked for more than 120 seconds. Aug 5 11:47:27 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:27 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:27 lola-7 kernel: ll_ost_io03_0 D 000000000000000f 0 24552 2 0x00000080 Aug 5 11:47:27 lola-7 kernel: ffff8803d33d77e0 0000000000000046 0000000000000000 ffff8803d33d77a4 Aug 5 11:47:27 lola-7 kernel: 0000001100000000 ffff88043fe83c00 000012f6632ad145 ffff88044e575a00 Aug 5 11:47:27 lola-7 kernel: 000000000000017f 000000010139b1dc ffff8803e974dad8 ffff8803d33d7fd8 Aug 5 11:47:27 lola-7 kernel: Call Trace: Aug 5 11:47:27 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:27 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:27 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:47:27 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:47:27 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:47:27 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:47:27 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:47:27 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:47:27 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:47:27 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:27 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:27 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:27 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:27 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:27 lola-7 kernel: INFO: task ll_ost03_004:24553 blocked for more than 120 seconds. Aug 5 11:47:27 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:27 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:27 lola-7 kernel: ll_ost03_004 D 000000000000001e 0 24553 2 0x00000080 Aug 5 11:47:27 lola-7 kernel: ffff8803ebce7940 0000000000000046 0000000000000080 0000000000000000 Aug 5 11:47:27 lola-7 kernel: ffff8803ebce78b0 ffffffffa03744f0 ffff8808175054e8 0000000000000013 Aug 5 11:47:27 lola-7 kernel: ffff8803ebce7a10 ffffffffa038d583 ffff8803e974d068 ffff8803ebce7fd8 Aug 5 11:47:27 lola-7 kernel: Call Trace: Aug 5 11:47:27 lola-7 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 11:47:27 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:27 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:27 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:47:27 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:47:27 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Aug 5 11:47:27 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:27 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:27 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:27 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:27 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:27 lola-7 kernel: INFO: task ll_ost_io02_005:24595 blocked for more than 120 seconds. Aug 5 11:47:27 lola-7 kernel: Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Aug 5 11:47:27 lola-7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 5 11:47:27 lola-7 kernel: ll_ost_io02_0 D 0000000000000019 0 24595 2 0x00000080 Aug 5 11:47:27 lola-7 kernel: ffff8803d180f7e0 0000000000000046 00000009d180f780 ffffffff0000000e Aug 5 11:47:27 lola-7 kernel: 00000011edde0258 0000000000000000 ffff8803d180f7c0 0000000000000050 Aug 5 11:47:27 lola-7 kernel: 0000000000000008 0000000000000000 ffff88041affbad8 ffff8803d180ffd8 Aug 5 11:47:27 lola-7 kernel: Call Trace: Aug 5 11:47:27 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 11:47:27 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:47:27 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:47:27 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:47:27 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:47:27 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 11:47:27 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:47:27 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:47:27 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:47:27 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:47:27 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:47:27 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:47:27 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:47:27 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:27 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:27 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:47:27 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:47:28 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:28 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:28 lola-7 kernel: LNet: Service thread pid 30861 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 11:47:28 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470422848.30861 Aug 5 11:47:29 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470422849.31115 Aug 5 11:47:30 lola-7 kernel: LNet: Service thread pid 30473 was inactive for 200.04s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 11:47:30 lola-7 kernel: LNet: Skipped 1 previous similar message Aug 5 11:47:30 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470422850.30435 Aug 5 11:47:32 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470422852.17193 Aug 5 11:47:47 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:47:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:47:47 lola-7 kernel: CPU 27 Aug 5 11:47:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:47:47 lola-7 kernel: Aug 5 11:47:47 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:47:47 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:47:47 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:47:47 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:47:47 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:47:47 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:47:47 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:47:47 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:47:47 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:47:47 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:47:47 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:47:47 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:47:47 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:47:47 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:47:47 lola-7 kernel: Stack: Aug 5 11:47:47 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:47:47 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:47:47 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:47:47 lola-7 kernel: Call Trace: Aug 5 11:47:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:47:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:47:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:47:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:47:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:47:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:47:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:47:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:47:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:47:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:47:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:47:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:47:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:47:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:47:47 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:47:47 lola-7 kernel: Call Trace: Aug 5 11:47:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:47:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:47:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:47:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:47:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:47:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:47:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:47:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:47:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:47:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:47:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:47:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:47:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:47:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:47:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:47:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:47:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:47:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:49:11 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:49:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:49:11 lola-7 kernel: CPU 27 Aug 5 11:49:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:49:11 lola-7 kernel: Aug 5 11:49:11 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:49:11 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:49:11 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:49:11 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:49:11 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:49:11 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:49:11 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:49:11 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:49:11 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:49:11 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:49:11 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:49:11 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:49:11 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:49:11 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:49:11 lola-7 kernel: Stack: Aug 5 11:49:11 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:49:11 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:49:11 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:49:11 lola-7 kernel: Call Trace: Aug 5 11:49:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:49:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:49:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:49:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:49:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:49:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:49:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:49:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:49:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:49:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:49:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:49:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:49:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:49:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:49:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:49:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:49:11 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:49:11 lola-7 kernel: Call Trace: Aug 5 11:49:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:49:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:49:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:49:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:49:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:49:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:49:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:49:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:49:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:49:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:49:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:49:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:49:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:49:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:49:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:49:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:49:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:49:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:50:35 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:50:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:50:35 lola-7 kernel: CPU 27 Aug 5 11:50:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:50:35 lola-7 kernel: Aug 5 11:50:35 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:50:35 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:50:35 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:50:35 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:50:35 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:50:35 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:50:35 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:50:35 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:50:35 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:50:35 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:50:35 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:50:35 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:50:35 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:50:35 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:50:35 lola-7 kernel: Stack: Aug 5 11:50:35 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:50:35 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:50:35 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:50:35 lola-7 kernel: Call Trace: Aug 5 11:50:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:50:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:50:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:50:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:50:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:50:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:50:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:50:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:50:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:50:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:50:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:50:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:50:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:50:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:50:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:50:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:50:35 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:50:35 lola-7 kernel: Call Trace: Aug 5 11:50:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:50:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:50:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:50:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:50:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:50:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:50:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:50:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:50:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:50:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:50:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:50:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:50:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:50:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:50:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:50:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:50:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:50:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:51:59 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:51:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:51:59 lola-7 kernel: CPU 27 Aug 5 11:51:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:51:59 lola-7 kernel: Aug 5 11:51:59 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:51:59 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:51:59 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:51:59 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:51:59 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:51:59 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:51:59 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:51:59 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:51:59 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:51:59 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:51:59 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:51:59 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:51:59 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:51:59 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:51:59 lola-7 kernel: Stack: Aug 5 11:51:59 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:51:59 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:51:59 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:51:59 lola-7 kernel: Call Trace: Aug 5 11:51:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:51:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:51:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:51:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:51:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:51:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:51:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:51:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:51:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:51:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:51:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:51:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:51:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:51:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:51:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:51:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:51:59 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:51:59 lola-7 kernel: Call Trace: Aug 5 11:51:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:51:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:51:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:51:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:51:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:51:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:51:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:51:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:51:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:51:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:51:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:51:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:51:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:51:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:51:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:51:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:51:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:51:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:52:00 lola-7 kernel: LNet: Service thread pid 30518 was inactive for 504.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 5 11:52:00 lola-7 kernel: LNet: Skipped 3 previous similar messages Aug 5 11:52:00 lola-7 kernel: Pid: 30518, comm: ll_ost03_043 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: Call Trace: Aug 5 11:52:00 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:52:00 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:52:00 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? osd_declare_quota+0x1c1/0x2d0 [osd_zfs] Aug 5 11:52:00 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:52:00 lola-7 kernel: [] ofd_precreate_objects+0xa2c/0x1910 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ? ofd_grant_create+0x2f3/0x4c0 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ofd_create_hdl+0x56e/0x2530 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:52:00 lola-7 kernel: [] ? reschedule_interrupt+0xe/0x20 Aug 5 11:52:00 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:52:00 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:52:00 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:52:00 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470423120.30518 Aug 5 11:52:00 lola-7 kernel: Pid: 24607, comm: ll_ost03_011 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: Call Trace: Aug 5 11:52:00 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:52:00 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:52:00 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:52:00 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:52:00 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Aug 5 11:52:00 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:52:00 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:52:00 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:52:00 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:52:00 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: Pid: 30504, comm: ll_ost03_036 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: Call Trace: Aug 5 11:52:00 lola-7 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 11:52:00 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:52:00 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:52:00 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:52:00 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:52:00 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Aug 5 11:52:00 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:52:00 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:52:00 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:52:00 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:52:00 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: Pid: 17179, comm: ll_ost03_001 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: Call Trace: Aug 5 11:52:00 lola-7 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Aug 5 11:52:00 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:52:00 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:52:00 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:52:00 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:52:00 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Aug 5 11:52:00 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:52:00 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:52:00 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:52:00 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:52:00 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: Pid: 30514, comm: ll_ost03_039 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: Call Trace: Aug 5 11:52:00 lola-7 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 11:52:00 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 11:52:00 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 11:52:00 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 11:52:00 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 11:52:00 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 11:52:00 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 11:52:00 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Aug 5 11:52:00 lola-7 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Aug 5 11:52:00 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:52:00 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:52:00 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 11:52:00 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 11:52:00 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:52:00 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:52:00 lola-7 kernel: Aug 5 11:52:00 lola-7 kernel: LNet: Service thread pid 24608 was inactive for 504.62s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 11:52:00 lola-7 kernel: LNet: Skipped 46 previous similar messages Aug 5 11:53:23 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:53:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:53:23 lola-7 kernel: CPU 27 Aug 5 11:53:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:53:23 lola-7 kernel: Aug 5 11:53:23 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:53:23 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:53:23 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:53:23 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:53:23 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:53:23 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:53:23 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:53:23 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:53:23 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:53:23 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:53:23 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:53:23 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:53:23 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:53:23 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:53:23 lola-7 kernel: Stack: Aug 5 11:53:23 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:53:23 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:53:23 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:53:23 lola-7 kernel: Call Trace: Aug 5 11:53:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:53:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:53:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:53:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:53:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:53:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:53:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:53:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:53:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:53:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:53:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:53:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:53:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:53:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:53:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:53:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:53:23 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:53:23 lola-7 kernel: Call Trace: Aug 5 11:53:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:53:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:53:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:53:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:53:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:53:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:53:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:53:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:53:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:53:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:53:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:53:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:53:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:53:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:53:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:53:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:53:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:53:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:53:31 lola-7 kernel: Lustre: 31110:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 11:53:31 lola-7 kernel: req@ffff880613bba450 x1541009645233552/t0(0) o4->9cff6698-27dd-142a-724f-49bd7985bea9@192.168.1.127@o2ib100:-1/-1 lens 608/448 e 20 to 0 dl 1470423216 ref 2 fl Interpret:/0/0 rc 0/0 Aug 5 11:53:31 lola-7 kernel: Lustre: 31328:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 11:53:31 lola-7 kernel: req@ffff88038d415450 x1541009460088592/t0(0) o4->cc7923f4-ad74-79f0-0ee2-358e1b6203e7@192.168.1.123@o2ib100:-1/-1 lens 608/448 e 18 to 0 dl 1470423216 ref 2 fl Interpret:/0/0 rc 0/0 Aug 5 11:53:31 lola-7 kernel: Lustre: 31328:0:(service.c:1335:ptlrpc_at_send_early_reply()) Skipped 62 previous similar messages Aug 5 11:53:37 lola-7 kernel: Lustre: soaked-OST0017: Client cc7923f4-ad74-79f0-0ee2-358e1b6203e7 (at 192.168.1.123@o2ib100) reconnecting Aug 5 11:53:37 lola-7 kernel: Lustre: soaked-OST0017: Connection restored to 30d2edae-f442-211c-f2f5-4cde9acc19f6 (at 192.168.1.113@o2ib100) Aug 5 11:53:37 lola-7 kernel: Lustre: Skipped 2 previous similar messages Aug 5 11:53:54 lola-7 kernel: Lustre: 30860:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 11:53:54 lola-7 kernel: req@ffff880824126050 x1541009385851856/t0(0) o4->c0d67e29-a13e-9b98-1a31-172a2686134a@192.168.1.121@o2ib100:-1/-1 lens 608/448 e 20 to 0 dl 1470423239 ref 2 fl Interpret:/0/0 rc 0/0 Aug 5 11:53:54 lola-7 kernel: Lustre: 30860:0:(service.c:1335:ptlrpc_at_send_early_reply()) Skipped 26 previous similar messages Aug 5 11:54:00 lola-7 kernel: Lustre: soaked-OST0017: Client c0d67e29-a13e-9b98-1a31-172a2686134a (at 192.168.1.121@o2ib100) reconnecting Aug 5 11:54:00 lola-7 kernel: Lustre: Skipped 6 previous similar messages Aug 5 11:54:03 lola-7 kernel: Lustre: 30860:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 11:54:03 lola-7 kernel: req@ffff88082412b050 x1541009566570064/t0(0) o4->96018fae-8d83-4949-5d4f-b8e2450e2ea9@192.168.1.129@o2ib100:-1/-1 lens 608/448 e 10 to 0 dl 1470423248 ref 2 fl Interpret:/0/0 rc 0/0 Aug 5 11:54:47 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:54:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:54:47 lola-7 kernel: CPU 27 Aug 5 11:54:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:54:47 lola-7 kernel: Aug 5 11:54:47 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:54:47 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:54:47 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:54:47 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:54:47 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:54:47 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:54:47 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:54:47 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:54:47 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:54:47 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:54:47 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:54:47 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:54:47 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:54:47 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:54:47 lola-7 kernel: Stack: Aug 5 11:54:47 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:54:47 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:54:47 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:54:47 lola-7 kernel: Call Trace: Aug 5 11:54:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:54:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:54:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:54:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:54:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:54:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:54:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:54:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:54:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:54:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:54:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:54:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:54:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:54:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:54:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:54:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:54:47 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:54:47 lola-7 kernel: Call Trace: Aug 5 11:54:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:54:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:54:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:54:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:54:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:54:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:54:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:54:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:54:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:54:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:54:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:54:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:54:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:54:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:54:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:54:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:54:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:54:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:54:50 lola-7 kernel: Lustre: soaked-OST0017: Client 8dfde31b-b4ea-ae0b-171e-8bd61886ac26 (at 192.168.1.124@o2ib100) reconnecting Aug 5 11:54:50 lola-7 kernel: Lustre: Skipped 2 previous similar messages Aug 5 11:56:11 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:56:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:56:11 lola-7 kernel: CPU 27 Aug 5 11:56:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:56:11 lola-7 kernel: Aug 5 11:56:11 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:56:11 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 11:56:11 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:56:11 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:56:11 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:56:11 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:56:11 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:56:11 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:56:11 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:56:11 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:56:11 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:56:11 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:56:11 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:56:11 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:56:11 lola-7 kernel: Stack: Aug 5 11:56:11 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:56:11 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:56:11 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:56:11 lola-7 kernel: Call Trace: Aug 5 11:56:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:56:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:56:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:56:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:56:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:56:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:56:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:56:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:56:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:56:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:56:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:56:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:56:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:56:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:56:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:56:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:56:11 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 11:56:11 lola-7 kernel: Call Trace: Aug 5 11:56:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:56:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:56:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:56:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:56:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:56:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:56:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:56:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:56:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:56:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:56:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:56:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:56:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:56:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:56:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:56:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:56:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:56:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:57:35 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:57:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:57:35 lola-7 kernel: CPU 27 Aug 5 11:57:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:57:35 lola-7 kernel: Aug 5 11:57:35 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:57:35 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:57:35 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:57:35 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:57:35 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:57:35 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:57:35 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:57:35 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:57:35 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:57:35 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:57:35 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:57:35 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:57:35 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:57:35 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:57:35 lola-7 kernel: Stack: Aug 5 11:57:35 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:57:35 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:57:35 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:57:35 lola-7 kernel: Call Trace: Aug 5 11:57:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:57:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:57:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:57:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:57:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:57:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:57:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:57:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:57:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:57:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:57:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:57:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:57:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:57:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:57:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:57:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:57:35 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:57:35 lola-7 kernel: Call Trace: Aug 5 11:57:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:57:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:57:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:57:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:57:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:57:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:57:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:57:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:57:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:57:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:57:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:57:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:57:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:57:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:57:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:57:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:57:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:57:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:58:59 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 11:58:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:58:59 lola-7 kernel: CPU 27 Aug 5 11:58:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 11:58:59 lola-7 kernel: Aug 5 11:58:59 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 11:58:59 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 11:58:59 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 11:58:59 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 11:58:59 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 11:58:59 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 11:58:59 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 11:58:59 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 11:58:59 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 11:58:59 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 11:58:59 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 11:58:59 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 11:58:59 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 11:58:59 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 11:58:59 lola-7 kernel: Stack: Aug 5 11:58:59 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 11:58:59 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 11:58:59 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 11:58:59 lola-7 kernel: Call Trace: Aug 5 11:58:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:58:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:58:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:58:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:58:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:58:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:58:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:58:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:58:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:58:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:58:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:58:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:58:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:58:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:58:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:58:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 11:58:59 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 11:58:59 lola-7 kernel: Call Trace: Aug 5 11:58:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 11:58:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 11:58:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 11:58:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 11:58:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 11:58:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 11:58:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 11:58:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 11:58:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 11:58:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 11:58:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 11:58:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 11:58:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 11:58:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 11:58:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 11:58:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 11:58:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 11:58:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:00:23 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:00:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:00:23 lola-7 kernel: CPU 27 Aug 5 12:00:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:00:23 lola-7 kernel: Aug 5 12:00:23 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:00:23 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:00:23 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:00:23 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:00:23 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:00:23 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:00:23 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:00:23 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:00:23 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:00:23 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:00:23 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:00:23 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:00:23 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:00:23 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:00:23 lola-7 kernel: Stack: Aug 5 12:00:23 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:00:23 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:00:23 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:00:23 lola-7 kernel: Call Trace: Aug 5 12:00:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:00:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:00:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:00:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:00:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:00:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:00:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:00:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:00:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:00:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:00:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:00:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:00:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:00:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:00:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:00:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:00:23 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:00:23 lola-7 kernel: Call Trace: Aug 5 12:00:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:00:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:00:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:00:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:00:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:00:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:00:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:00:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:00:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:00:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:00:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:00:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:00:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:00:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:00:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:00:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:00:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:00:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:01:47 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:01:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:01:47 lola-7 kernel: CPU 27 Aug 5 12:01:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:01:47 lola-7 kernel: Aug 5 12:01:47 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:01:47 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:01:47 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:01:47 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:01:47 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:01:47 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:01:47 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:01:47 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:01:47 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:01:47 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:01:47 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:01:47 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:01:47 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:01:47 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:01:47 lola-7 kernel: Stack: Aug 5 12:01:47 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:01:47 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:01:47 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:01:47 lola-7 kernel: Call Trace: Aug 5 12:01:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:01:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:01:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:01:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:01:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:01:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:01:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:01:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:01:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:01:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:01:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:01:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:01:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:01:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:01:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:01:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:01:47 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:01:47 lola-7 kernel: Call Trace: Aug 5 12:01:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:01:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:01:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:01:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:01:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:01:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:01:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:01:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:01:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:01:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:01:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:01:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:01:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:01:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:01:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:01:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:01:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:01:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:03:11 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:03:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:03:11 lola-7 kernel: CPU 27 Aug 5 12:03:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:03:11 lola-7 kernel: Aug 5 12:03:11 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:03:11 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x7/0x20 Aug 5 12:03:11 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:03:11 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:03:11 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:03:11 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:03:11 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:03:11 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:03:11 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:03:11 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:03:11 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:03:11 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:03:11 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:03:11 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:03:11 lola-7 kernel: Stack: Aug 5 12:03:11 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:03:11 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:03:11 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:03:11 lola-7 kernel: Call Trace: Aug 5 12:03:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:03:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:03:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:03:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:03:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:03:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:03:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:03:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:03:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:03:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:03:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:03:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:03:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:03:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:03:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:03:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:03:11 lola-7 kernel: Code: d8 3f 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 90 81 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff Aug 5 12:03:11 lola-7 kernel: Call Trace: Aug 5 12:03:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:03:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:03:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:03:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:03:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:03:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:03:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:03:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:03:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:03:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:03:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:03:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:03:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:03:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:03:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:03:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:03:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:03:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:03:32 lola-7 kernel: Lustre: 31330:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 12:03:32 lola-7 kernel: req@ffff8803f986a050 x1541019794118224/t0(0) o4->c66965dc-2291-0fd5-8b66-8301653caa82@192.168.1.126@o2ib100:-1/-1 lens 608/448 e 0 to 0 dl 1470423817 ref 2 fl Interpret:/2/0 rc 0/0 Aug 5 12:03:32 lola-7 kernel: Lustre: 31330:0:(service.c:1335:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages Aug 5 12:03:38 lola-7 kernel: Lustre: soaked-OST0017: Client 3cc8f03d-59b0-76b4-60ec-19a13d95af15 (at 192.168.1.122@o2ib100) reconnecting Aug 5 12:03:38 lola-7 kernel: Lustre: soaked-OST0017: Connection restored to 3cc8f03d-59b0-76b4-60ec-19a13d95af15 (at 192.168.1.122@o2ib100) Aug 5 12:03:38 lola-7 kernel: Lustre: Skipped 12 previous similar messages Aug 5 12:03:55 lola-7 kernel: Lustre: 31371:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 12:03:55 lola-7 kernel: req@ffff880824bfc050 x1541009385851856/t0(0) o4->c0d67e29-a13e-9b98-1a31-172a2686134a@192.168.1.121@o2ib100:-1/-1 lens 608/448 e 0 to 0 dl 1470423840 ref 2 fl Interpret:/2/0 rc 0/0 Aug 5 12:03:55 lola-7 kernel: Lustre: 31371:0:(service.c:1335:ptlrpc_at_send_early_reply()) Skipped 71 previous similar messages Aug 5 12:04:35 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:04:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:04:35 lola-7 kernel: CPU 27 Aug 5 12:04:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:04:35 lola-7 kernel: Aug 5 12:04:35 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:04:35 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:04:35 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:04:35 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:04:35 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:04:35 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:04:35 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:04:35 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:04:35 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:04:35 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:04:35 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:04:35 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:04:35 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:04:35 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:04:35 lola-7 kernel: Stack: Aug 5 12:04:35 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:04:35 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:04:35 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:04:35 lola-7 kernel: Call Trace: Aug 5 12:04:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:04:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:04:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:04:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:04:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:04:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:04:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:04:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:04:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:04:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:04:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:04:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:04:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:04:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:04:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:04:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:04:35 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:04:35 lola-7 kernel: Call Trace: Aug 5 12:04:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:04:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:04:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:04:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:04:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:04:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:04:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:04:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:04:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:04:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:04:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:04:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:04:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:04:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:04:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:04:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:04:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:04:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:04:45 lola-7 kernel: Lustre: 31376:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 12:04:45 lola-7 kernel: req@ffff8807cc95c450 x1541019923925360/t0(0) o4->8dfde31b-b4ea-ae0b-171e-8bd61886ac26@192.168.1.124@o2ib100:-1/-1 lens 608/448 e 0 to 0 dl 1470423890 ref 2 fl Interpret:/2/0 rc 0/0 Aug 5 12:04:45 lola-7 kernel: Lustre: 31376:0:(service.c:1335:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Aug 5 12:04:51 lola-7 kernel: Lustre: soaked-OST0017: Client 8dfde31b-b4ea-ae0b-171e-8bd61886ac26 (at 192.168.1.124@o2ib100) reconnecting Aug 5 12:04:51 lola-7 kernel: Lustre: Skipped 11 previous similar messages Aug 5 12:05:59 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:05:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:05:59 lola-7 kernel: CPU 27 Aug 5 12:05:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:05:59 lola-7 kernel: Aug 5 12:05:59 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:05:59 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:05:59 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:05:59 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:05:59 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:05:59 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:05:59 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:05:59 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:05:59 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:05:59 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:05:59 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:05:59 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:05:59 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:05:59 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:05:59 lola-7 kernel: Stack: Aug 5 12:05:59 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:05:59 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:05:59 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:05:59 lola-7 kernel: Call Trace: Aug 5 12:05:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:05:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:05:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:05:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:05:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:05:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:05:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:05:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:05:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:05:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:05:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:05:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:05:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:05:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:05:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:05:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:05:59 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:05:59 lola-7 kernel: Call Trace: Aug 5 12:05:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:05:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:05:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:05:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:05:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:05:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:05:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:05:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:05:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:05:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:05:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:05:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:05:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:05:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:05:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:05:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:05:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:05:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:06:07 lola-7 kernel: Lustre: 30485:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Aug 5 12:06:07 lola-7 kernel: req@ffff880754130850 x1541788331101312/t0(0) o5->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 432/432 e 0 to 0 dl 1470423972 ref 2 fl Interpret:/0/0 rc 0/0 Aug 5 12:06:07 lola-7 kernel: Lustre: 30485:0:(service.c:1335:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages Aug 5 12:07:23 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:07:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:07:23 lola-7 kernel: CPU 27 Aug 5 12:07:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:07:23 lola-7 kernel: Aug 5 12:07:23 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:07:23 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x7/0x20 Aug 5 12:07:23 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:07:23 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:07:23 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:07:23 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:07:23 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:07:23 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:07:23 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:07:23 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:07:23 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:07:23 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:07:23 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:07:23 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:07:23 lola-7 kernel: Stack: Aug 5 12:07:23 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:07:23 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:07:23 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:07:23 lola-7 kernel: Call Trace: Aug 5 12:07:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:07:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:07:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:07:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:07:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:07:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:07:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:07:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:07:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:07:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:07:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:07:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:07:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:07:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:07:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:07:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:07:23 lola-7 kernel: Code: d8 3f 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 90 81 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff Aug 5 12:07:23 lola-7 kernel: Call Trace: Aug 5 12:07:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:07:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:07:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:07:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:07:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:07:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:07:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:07:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:07:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:07:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:07:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:07:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:07:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:07:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:07:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:07:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:07:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:07:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:08:47 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:08:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:08:47 lola-7 kernel: CPU 27 Aug 5 12:08:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:08:47 lola-7 kernel: Aug 5 12:08:47 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:08:47 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:08:47 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:08:47 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:08:47 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:08:47 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:08:47 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:08:47 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:08:47 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:08:47 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:08:47 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:08:47 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:08:47 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:08:47 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:08:47 lola-7 kernel: Stack: Aug 5 12:08:47 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:08:47 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:08:47 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:08:47 lola-7 kernel: Call Trace: Aug 5 12:08:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:08:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:08:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:08:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:08:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:08:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:08:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:08:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:08:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:08:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:08:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:08:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:08:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:08:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:08:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:08:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:08:47 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:08:47 lola-7 kernel: Call Trace: Aug 5 12:08:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:08:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:08:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:08:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:08:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:08:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:08:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:08:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:08:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:08:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:08:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:08:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:08:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:08:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:08:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:08:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:08:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:08:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:10:11 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:10:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:10:11 lola-7 kernel: CPU 27 Aug 5 12:10:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:10:11 lola-7 kernel: Aug 5 12:10:11 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:10:11 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:10:11 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:10:11 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:10:11 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:10:11 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:10:11 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:10:11 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:10:11 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:10:11 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:10:11 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:10:11 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:10:11 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:10:11 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:10:11 lola-7 kernel: Stack: Aug 5 12:10:11 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:10:11 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:10:11 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:10:11 lola-7 kernel: Call Trace: Aug 5 12:10:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:10:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:10:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:10:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:10:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:10:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:10:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:10:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:10:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:10:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:10:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:10:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:10:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:10:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:10:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:10:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:10:11 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:10:11 lola-7 kernel: Call Trace: Aug 5 12:10:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:10:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:10:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:10:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:10:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:10:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:10:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:10:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:10:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:10:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:10:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:10:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:10:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:10:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:10:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:10:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:10:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:10:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:11:35 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:11:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:11:35 lola-7 kernel: CPU 27 Aug 5 12:11:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:11:35 lola-7 kernel: Aug 5 12:11:35 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:11:35 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:11:35 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:11:35 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:11:35 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:11:35 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:11:35 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:11:35 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:11:35 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:11:35 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:11:35 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:11:35 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:11:35 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:11:35 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:11:35 lola-7 kernel: Stack: Aug 5 12:11:35 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:11:35 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:11:35 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:11:35 lola-7 kernel: Call Trace: Aug 5 12:11:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:11:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:11:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:11:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:11:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:11:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:11:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:11:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:11:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:11:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:11:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:11:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:11:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:11:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:11:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:11:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:11:35 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:11:35 lola-7 kernel: Call Trace: Aug 5 12:11:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:11:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:11:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:11:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:11:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:11:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:11:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:11:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:11:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:11:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:11:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:11:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:11:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:11:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:11:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:11:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:11:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:11:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:12:59 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:12:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:12:59 lola-7 kernel: CPU 27 Aug 5 12:12:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:12:59 lola-7 kernel: Aug 5 12:12:59 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:12:59 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:12:59 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:12:59 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:12:59 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:12:59 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:12:59 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:12:59 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:12:59 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:12:59 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:12:59 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:12:59 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:12:59 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:12:59 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:12:59 lola-7 kernel: Stack: Aug 5 12:12:59 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:12:59 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:12:59 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:12:59 lola-7 kernel: Call Trace: Aug 5 12:12:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:12:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:12:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:12:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:12:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:12:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:12:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:12:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:12:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:12:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:12:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:12:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:12:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:12:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:12:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:12:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:12:59 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:12:59 lola-7 kernel: Call Trace: Aug 5 12:12:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:12:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:12:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:12:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:12:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:12:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:12:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:12:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:12:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:12:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:12:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:12:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:12:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:12:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:12:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:12:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:12:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:12:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:13:33 lola-7 kernel: Lustre: 30885:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 12:13:33 lola-7 kernel: req@ffff8801dc2a6050 x1541009460083552/t0(0) o4->cc7923f4-ad74-79f0-0ee2-358e1b6203e7@192.168.1.123@o2ib100:-1/-1 lens 608/448 e 0 to 0 dl 1470424418 ref 2 fl Interpret:/2/0 rc 0/0 Aug 5 12:13:37 lola-7 kernel: LNet: Service thread pid 30470 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 5 12:13:37 lola-7 kernel: LNet: Skipped 4 previous similar messages Aug 5 12:13:37 lola-7 kernel: Pid: 30470, comm: ll_ost_io00_036 Aug 5 12:13:37 lola-7 kernel: Aug 5 12:13:37 lola-7 kernel: Call Trace: Aug 5 12:13:37 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:13:37 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:13:37 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:13:37 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:13:37 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:13:37 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:13:37 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 12:13:37 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:13:37 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 12:13:37 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 12:13:37 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:13:37 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:13:37 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:13:37 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:13:37 lola-7 kernel: Aug 5 12:13:37 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470424417.30470 Aug 5 12:13:37 lola-7 kernel: Pid: 31403, comm: ll_ost_io01_062 Aug 5 12:13:37 lola-7 kernel: Aug 5 12:13:37 lola-7 kernel: Call Trace: Aug 5 12:13:37 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:13:37 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:13:37 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:13:37 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:13:37 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:13:37 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:13:37 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:13:37 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:13:37 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:13:37 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:13:37 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:13:37 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:13:37 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:13:37 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:13:37 lola-7 kernel: Aug 5 12:13:37 lola-7 kernel: Pid: 31373, comm: ll_ost_io03_083 Aug 5 12:13:37 lola-7 kernel: Aug 5 12:13:37 lola-7 kernel: Call Trace: Aug 5 12:13:37 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 12:13:37 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:13:37 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:13:37 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:13:37 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:13:37 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:13:37 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:13:37 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:13:37 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:13:37 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:13:37 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 12:13:37 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 12:13:37 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:13:37 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:13:37 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:13:37 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:13:37 lola-7 kernel: Aug 5 12:13:37 lola-7 kernel: LNet: Service thread pid 30800 was inactive for 1200.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 5 12:13:37 lola-7 kernel: LNet: Skipped 2 previous similar messages Aug 5 12:13:37 lola-7 kernel: Pid: 30800, comm: ll_ost_io03_028 Aug 5 12:13:37 lola-7 kernel: Aug 5 12:13:37 lola-7 kernel: Call Trace: Aug 5 12:13:37 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:13:37 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:13:37 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:13:37 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:13:37 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:13:37 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:13:37 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:13:37 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 12:13:37 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:13:37 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:13:37 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:13:37 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:13:37 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:13:37 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:13:37 lola-7 kernel: Aug 5 12:13:37 lola-7 kernel: Pid: 30879, comm: ll_ost_io03_056 Aug 5 12:13:37 lola-7 kernel: Aug 5 12:13:37 lola-7 kernel: Call Trace: Aug 5 12:13:37 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 12:13:37 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:13:37 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:13:37 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:13:37 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:13:37 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:13:38 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:13:38 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:13:38 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:13:38 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:13:38 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:13:38 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:13:38 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:13:38 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:13:38 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 12:13:38 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:13:38 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:13:38 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:13:38 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:13:38 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 12:13:38 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 12:13:38 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:13:38 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:13:38 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:13:38 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:13:38 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:13:38 lola-7 kernel: Aug 5 12:13:38 lola-7 kernel: LNet: Service thread pid 30437 was inactive for 1200.05s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 12:13:38 lola-7 kernel: LNet: Skipped 3 previous similar messages Aug 5 12:13:39 lola-7 kernel: Lustre: soaked-OST0017: Client cc7923f4-ad74-79f0-0ee2-358e1b6203e7 (at 192.168.1.123@o2ib100) reconnecting Aug 5 12:13:39 lola-7 kernel: Lustre: soaked-OST0017: Connection restored to cc7923f4-ad74-79f0-0ee2-358e1b6203e7 (at 192.168.1.123@o2ib100) Aug 5 12:13:39 lola-7 kernel: Lustre: Skipped 12 previous similar messages Aug 5 12:14:00 lola-7 kernel: LNet: Service thread pid 30426 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 12:14:00 lola-7 kernel: LNet: Skipped 67 previous similar messages Aug 5 12:14:00 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470424440.30426 Aug 5 12:14:09 lola-7 kernel: LNet: Service thread pid 31349 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 12:14:09 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470424449.31349 Aug 5 12:14:10 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470424450.31364 Aug 5 12:14:23 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:14:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:14:23 lola-7 kernel: CPU 27 Aug 5 12:14:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:14:23 lola-7 kernel: Aug 5 12:14:23 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:14:23 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:14:23 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:14:23 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:14:23 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:14:23 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:14:23 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:14:23 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:14:23 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:14:23 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:14:23 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:14:23 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:14:23 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:14:23 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:14:23 lola-7 kernel: Stack: Aug 5 12:14:23 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:14:23 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:14:23 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:14:23 lola-7 kernel: Call Trace: Aug 5 12:14:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:14:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:14:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:14:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:14:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:14:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:14:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:14:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:14:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:14:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:14:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:14:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:14:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:14:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:14:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:14:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:14:23 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:14:23 lola-7 kernel: Call Trace: Aug 5 12:14:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:14:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:14:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:14:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:14:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:14:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:14:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:14:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:14:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:14:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:14:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:14:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:14:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:14:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:14:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:14:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:14:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:14:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:14:50 lola-7 kernel: LNet: Service thread pid 31399 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 12:14:50 lola-7 kernel: LNet: Skipped 3 previous similar messages Aug 5 12:14:50 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470424490.31399 Aug 5 12:15:47 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:15:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:15:47 lola-7 kernel: CPU 27 Aug 5 12:15:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:15:47 lola-7 kernel: Aug 5 12:15:47 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:15:47 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:15:47 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:15:47 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:15:47 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:15:47 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:15:47 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:15:47 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:15:47 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:15:47 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:15:47 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:15:47 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:15:47 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:15:47 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:15:47 lola-7 kernel: Stack: Aug 5 12:15:47 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:15:47 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:15:47 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:15:47 lola-7 kernel: Call Trace: Aug 5 12:15:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:15:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:15:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:15:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:15:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:15:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:15:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:15:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:15:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:15:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:15:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:15:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:15:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:15:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:15:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:15:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:15:47 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:15:47 lola-7 kernel: Call Trace: Aug 5 12:15:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:15:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:15:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:15:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:15:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:15:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:15:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:15:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:15:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:15:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:15:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:15:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:15:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:15:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:15:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:15:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:15:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:15:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:17:11 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:17:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:17:11 lola-7 kernel: CPU 27 Aug 5 12:17:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:17:11 lola-7 kernel: Aug 5 12:17:11 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:17:11 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:17:11 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:17:11 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:17:11 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:17:11 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:17:11 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:17:11 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:17:11 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:17:11 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:17:11 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:17:11 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:17:11 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:17:11 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:17:11 lola-7 kernel: Stack: Aug 5 12:17:11 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:17:11 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:17:11 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:17:11 lola-7 kernel: Call Trace: Aug 5 12:17:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:17:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:17:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:17:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:17:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:17:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:17:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:17:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:17:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:17:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:17:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:17:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:17:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:17:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:17:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:17:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:17:11 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:17:11 lola-7 kernel: Call Trace: Aug 5 12:17:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:17:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:17:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:17:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:17:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:17:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:17:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:17:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:17:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:17:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:17:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:17:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:17:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:17:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:17:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:17:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:17:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:17:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:18:35 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:18:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:18:35 lola-7 kernel: CPU 27 Aug 5 12:18:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:18:35 lola-7 kernel: Aug 5 12:18:35 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:18:35 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:18:35 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:18:35 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:18:35 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:18:35 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:18:35 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:18:35 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:18:35 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:18:35 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:18:35 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:18:35 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:18:35 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:18:35 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:18:35 lola-7 kernel: Stack: Aug 5 12:18:35 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:18:35 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:18:35 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:18:35 lola-7 kernel: Call Trace: Aug 5 12:18:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:18:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:18:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:18:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:18:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:18:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:18:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:18:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:18:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:18:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:18:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:18:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:18:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:18:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:18:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:18:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:18:35 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:18:35 lola-7 kernel: Call Trace: Aug 5 12:18:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:18:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:18:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:18:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:18:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:18:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:18:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:18:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:18:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:18:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:18:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:18:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:18:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:18:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:18:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:18:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:18:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:18:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:18:43 lola-7 kernel: Lustre: 30483:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Aug 5 12:18:43 lola-7 kernel: req@ffff880751dd7450 x1541788333336976/t0(0) o5->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 432/432 e 0 to 0 dl 1470424728 ref 2 fl Interpret:/0/0 rc 0/0 Aug 5 12:18:43 lola-7 kernel: Lustre: 30483:0:(service.c:1335:ptlrpc_at_send_early_reply()) Skipped 85 previous similar messages Aug 5 12:19:34 lola-7 kernel: Lustre: 17132:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1470424767/real 0] req@ffff8805fce550c0 x1541832705589392/t0(0) o400->soaked-MDT0001-lwp-OST0005@192.168.1.109@o2ib10:12/10 lens 224/224 e 0 to 1 dl 1470424774 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 5 12:19:34 lola-7 kernel: Lustre: soaked-MDT0001-lwp-OST0011: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Aug 5 12:19:34 lola-7 kernel: Lustre: Skipped 4 previous similar messages Aug 5 12:19:34 lola-7 kernel: Lustre: 17132:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Aug 5 12:19:59 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:19:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:19:59 lola-7 kernel: CPU 27 Aug 5 12:19:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:19:59 lola-7 kernel: Aug 5 12:19:59 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:19:59 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:19:59 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:19:59 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:19:59 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:19:59 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:19:59 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:19:59 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:19:59 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:19:59 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:19:59 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:19:59 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:19:59 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:19:59 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:19:59 lola-7 kernel: Stack: Aug 5 12:19:59 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:19:59 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:19:59 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:19:59 lola-7 kernel: Call Trace: Aug 5 12:19:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:19:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:19:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:19:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:19:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:19:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:19:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:19:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:19:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:19:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:19:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:19:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:19:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:19:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:19:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:19:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:19:59 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:19:59 lola-7 kernel: Call Trace: Aug 5 12:19:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:19:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:19:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:19:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:19:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:19:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:19:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:19:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:19:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:19:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:19:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:19:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:19:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:19:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:19:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:19:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:19:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:19:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:20:35 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1470424824/real 0] req@ffff88060af56c80 x1541832705590192/t0(0) o38->soaked-MDT0001-lwp-OST0017@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1470424835 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 5 12:20:35 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Aug 5 12:21:23 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:21:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:21:23 lola-7 kernel: CPU 27 Aug 5 12:21:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:21:23 lola-7 kernel: Aug 5 12:21:23 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:21:23 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:21:23 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:21:23 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:21:23 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:21:23 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:21:23 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:21:23 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:21:23 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:21:23 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:21:23 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:21:23 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:21:23 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:21:23 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:21:23 lola-7 kernel: Stack: Aug 5 12:21:23 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:21:23 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:21:23 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:21:23 lola-7 kernel: Call Trace: Aug 5 12:21:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:21:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:21:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:21:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:21:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:21:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:21:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:21:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:21:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:21:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:21:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:21:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:21:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:21:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:21:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:21:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:21:23 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:21:23 lola-7 kernel: Call Trace: Aug 5 12:21:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:21:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:21:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:21:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:21:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:21:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:21:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:21:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:21:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:21:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:21:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:21:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:21:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:21:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:21:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:21:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:21:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:21:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:22:20 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1470424924/real 0] req@ffff8806010170c0 x1541832705591152/t0(0) o38->soaked-MDT0001-lwp-OST0017@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1470424940 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 5 12:22:20 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 5 12:22:47 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:22:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:22:47 lola-7 kernel: CPU 27 Aug 5 12:22:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:22:47 lola-7 kernel: Aug 5 12:22:47 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:22:47 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:22:47 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:22:47 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:22:47 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:22:47 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:22:47 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:22:47 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:22:47 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:22:47 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:22:47 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:22:47 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:22:47 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:22:47 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:22:47 lola-7 kernel: Stack: Aug 5 12:22:47 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:22:47 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:22:47 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:22:47 lola-7 kernel: Call Trace: Aug 5 12:22:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:22:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:22:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:22:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:22:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:22:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:22:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:22:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:22:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:22:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:22:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:22:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:22:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:22:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:22:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:22:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:22:47 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:22:47 lola-7 kernel: Call Trace: Aug 5 12:22:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:22:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:22:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:22:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:22:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:22:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:22:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:22:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:22:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:22:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:22:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:22:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:22:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:22:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:22:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:22:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:22:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:22:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:22:48 lola-7 kernel: Lustre: soaked-OST0017: haven't heard from client soaked-MDT0001-mdtlov_UUID (at 192.168.1.109@o2ib10) in 228 seconds. I think it's dead, and I am evicting it. exp ffff8807a8c2b800, cur 1470424968 expire 1470424818 last 1470424740 Aug 5 12:22:48 lola-7 kernel: Lustre: Skipped 7 previous similar messages Aug 5 12:23:34 lola-7 kernel: Lustre: 31312:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 12:23:34 lola-7 kernel: req@ffff88038d415c50 x1541849347171456/t0(0) o4->50fa8a2c-9e9a-5b16-a82f-1d1ac7f58c89@192.168.1.113@o2ib100:-1/-1 lens 608/0 e 0 to 0 dl 1470425019 ref 2 fl New:/2/ffffffff rc 0/-1 Aug 5 12:23:38 lola-7 kernel: LNet: Service thread pid 24585 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 5 12:23:38 lola-7 kernel: LNet: Skipped 1 previous similar message Aug 5 12:23:38 lola-7 kernel: Pid: 24585, comm: ll_ost_io02_003 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: Call Trace: Aug 5 12:23:38 lola-7 kernel: [] rwsem_down_failed_common+0x95/0x1d0 Aug 5 12:23:38 lola-7 kernel: [] rwsem_down_read_failed+0x26/0x30 Aug 5 12:23:38 lola-7 kernel: [] call_rwsem_down_read_failed+0x14/0x30 Aug 5 12:23:38 lola-7 kernel: [] ? down_read+0x24/0x30 Aug 5 12:23:38 lola-7 kernel: [] osd_attr_get+0x77/0x2e0 [osd_zfs] Aug 5 12:23:38 lola-7 kernel: [] ofd_attr_handle_ugid+0x161/0x430 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_write_attr_set+0xad/0x8b0 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw_write+0x22e/0xfa0 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ? ofd_fmd_find_nolock+0xad/0xd0 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:23:38 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:23:38 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 12:23:38 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:23:38 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:23:38 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:23:38 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:23:38 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425018.24585 Aug 5 12:23:38 lola-7 kernel: Pid: 30833, comm: ll_ost_io01_038 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: Call Trace: Aug 5 12:23:38 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:23:38 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:23:38 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:23:38 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:23:38 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:23:38 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:23:38 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:23:38 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:23:38 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:23:38 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:23:38 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:23:38 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:23:38 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:23:38 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:23:38 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:23:38 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:23:38 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:23:38 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: Pid: 31113, comm: ll_ost_io02_042 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: Call Trace: Aug 5 12:23:38 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:23:38 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:23:38 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:23:38 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:23:38 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:23:38 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:23:38 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:23:38 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:23:38 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:23:38 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:23:38 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:23:38 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:23:38 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:23:38 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:23:38 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:23:38 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:23:38 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:23:38 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: Pid: 30424, comm: ll_ost_io03_011 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: Call Trace: Aug 5 12:23:38 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:23:38 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:23:38 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:23:38 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:23:38 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:23:38 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:23:38 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:23:38 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:23:38 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:23:38 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:23:38 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:23:38 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:23:38 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:23:38 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:23:38 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:23:38 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:23:38 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:23:38 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: Pid: 31348, comm: ll_ost_io03_077 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: Call Trace: Aug 5 12:23:38 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:23:38 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:23:38 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:23:38 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:23:38 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:23:38 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:23:38 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:23:38 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:23:38 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:23:38 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:23:38 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:23:38 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 12:23:38 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:23:38 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:23:38 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:23:38 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:23:38 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:23:38 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:23:38 lola-7 kernel: Aug 5 12:23:38 lola-7 kernel: LNet: Service thread pid 31306 was inactive for 1200.06s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 12:23:38 lola-7 kernel: LNet: Skipped 8 previous similar messages Aug 5 12:23:40 lola-7 kernel: Lustre: soaked-OST0017: Client d3d6a29a-0595-e79b-060a-a9c1839ba326 (at 192.168.1.120@o2ib100) reconnecting Aug 5 12:23:40 lola-7 kernel: Lustre: Skipped 12 previous similar messages Aug 5 12:23:40 lola-7 kernel: Lustre: soaked-OST0017: Connection restored to d3d6a29a-0595-e79b-060a-a9c1839ba326 (at 192.168.1.120@o2ib100) Aug 5 12:23:40 lola-7 kernel: Lustre: Skipped 11 previous similar messages Aug 5 12:24:01 lola-7 kernel: LNet: Service thread pid 31371 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 12:24:01 lola-7 kernel: LNet: Skipped 66 previous similar messages Aug 5 12:24:01 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425041.31371 Aug 5 12:24:05 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1470425024/real 0] req@ffff8805fd16fcc0 x1541832705748160/t0(0) o38->soaked-MDT0001-lwp-OST000b@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1470425045 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 5 12:24:05 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 5 12:24:10 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425050.31317 Aug 5 12:24:11 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:24:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:24:11 lola-7 kernel: CPU 27 Aug 5 12:24:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:24:11 lola-7 kernel: Aug 5 12:24:11 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:24:11 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:24:11 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:24:11 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:24:11 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:24:11 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:24:11 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:24:11 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:24:11 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:24:11 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:24:11 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:24:11 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:24:11 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:24:11 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:24:11 lola-7 kernel: Stack: Aug 5 12:24:11 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:24:11 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:24:11 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:24:11 lola-7 kernel: Call Trace: Aug 5 12:24:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:24:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:24:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:24:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:24:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:24:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:24:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:24:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:24:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:24:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:24:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:24:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:24:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:24:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:24:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:24:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:24:11 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:24:11 lola-7 kernel: Call Trace: Aug 5 12:24:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:24:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:24:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:24:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:24:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:24:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:24:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:24:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:24:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:24:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:24:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:24:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:24:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:24:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:24:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:24:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:24:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:24:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:24:12 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425052.30864 Aug 5 12:24:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 384 Aug 5 12:24:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 387 Aug 5 12:24:51 lola-7 kernel: LNet: Service thread pid 30803 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 12:24:51 lola-7 kernel: LNet: Skipped 4 previous similar messages Aug 5 12:24:51 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425091.30803 Aug 5 12:25:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 390 Aug 5 12:25:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 394 Aug 5 12:25:24 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1470425124/real 1470425124] req@ffff8806cdd32980 x1541832705749120/t0(0) o38->soaked-MDT0001-lwp-OST000b@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1470425150 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 5 12:25:24 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 5 12:25:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 396 Aug 5 12:25:35 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:25:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:25:35 lola-7 kernel: CPU 27 Aug 5 12:25:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:25:35 lola-7 kernel: Aug 5 12:25:35 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:25:35 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:25:35 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:25:35 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:25:35 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:25:35 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:25:35 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:25:35 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:25:35 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:25:35 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:25:35 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:25:35 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:25:35 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:25:35 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:25:35 lola-7 kernel: Stack: Aug 5 12:25:35 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:25:35 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:25:35 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:25:35 lola-7 kernel: Call Trace: Aug 5 12:25:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:25:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:25:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:25:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:25:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:25:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:25:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:25:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:25:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:25:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:25:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:25:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:25:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:25:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:25:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:25:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:25:35 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:25:35 lola-7 kernel: Call Trace: Aug 5 12:25:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:25:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:25:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:25:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:25:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:25:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:25:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:25:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:25:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:25:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:25:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:25:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:25:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:25:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:25:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:25:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:25:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:25:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:25:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 398 Aug 5 12:26:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 400 Aug 5 12:26:13 lola-7 kernel: LNet: Service thread pid 30503 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 12:26:13 lola-7 kernel: LNet: Skipped 8 previous similar messages Aug 5 12:26:13 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425173.30503 Aug 5 12:26:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 401 Aug 5 12:26:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 402 Aug 5 12:26:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 402 Aug 5 12:26:59 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:26:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:26:59 lola-7 kernel: CPU 27 Aug 5 12:26:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:26:59 lola-7 kernel: Aug 5 12:26:59 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:26:59 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:26:59 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:26:59 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:26:59 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:26:59 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:26:59 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:26:59 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:26:59 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:26:59 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:26:59 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:26:59 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:26:59 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:26:59 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:26:59 lola-7 kernel: Stack: Aug 5 12:26:59 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:26:59 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:26:59 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:26:59 lola-7 kernel: Call Trace: Aug 5 12:26:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:26:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:26:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:26:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:26:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:26:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:26:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:26:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:26:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:26:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:26:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:26:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:26:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:26:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:26:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:26:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:26:59 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:26:59 lola-7 kernel: Call Trace: Aug 5 12:26:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:26:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:26:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:26:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:26:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:26:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:26:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:26:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:26:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:26:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:26:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:26:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:26:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:26:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:26:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:26:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:26:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:26:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:27:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 403 Aug 5 12:27:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 403 Aug 5 12:27:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 404 Aug 5 12:27:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 404 Aug 5 12:27:54 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1470425274/real 1470425274] req@ffff8806bdce9980 x1541832705906640/t0(0) o38->soaked-MDT0001-lwp-OST000b@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1470425315 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 5 12:27:54 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Aug 5 12:28:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 405 Aug 5 12:28:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 405 Aug 5 12:28:23 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:28:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:28:23 lola-7 kernel: CPU 27 Aug 5 12:28:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:28:23 lola-7 kernel: Aug 5 12:28:23 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:28:23 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:28:23 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:28:23 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:28:23 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:28:23 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:28:23 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:28:23 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:28:23 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:28:23 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:28:23 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:28:23 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:28:23 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:28:23 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:28:23 lola-7 kernel: Stack: Aug 5 12:28:23 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:28:23 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:28:23 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:28:23 lola-7 kernel: Call Trace: Aug 5 12:28:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:28:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:28:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:28:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:28:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:28:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:28:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:28:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:28:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:28:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:28:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:28:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:28:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:28:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:28:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:28:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:28:23 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:28:23 lola-7 kernel: Call Trace: Aug 5 12:28:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:28:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:28:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:28:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:28:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:28:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:28:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:28:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:28:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:28:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:28:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:28:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:28:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:28:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:28:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:28:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:28:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:28:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:28:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 405 Aug 5 12:28:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 405 Aug 5 12:29:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 405 Aug 5 12:29:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 406 Aug 5 12:29:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 406 Aug 5 12:29:47 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 68s! [ll_ost_io02_012:30458] Aug 5 12:29:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:29:47 lola-7 kernel: CPU 27 Aug 5 12:29:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:29:47 lola-7 kernel: Aug 5 12:29:47 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:29:47 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:29:47 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:29:47 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:29:47 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:29:47 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:29:47 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:29:47 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:29:47 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:29:47 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:29:47 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:29:47 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:29:47 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:29:47 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:29:47 lola-7 kernel: Stack: Aug 5 12:29:47 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:29:47 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:29:47 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:29:47 lola-7 kernel: Call Trace: Aug 5 12:29:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:29:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:29:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:29:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:29:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:29:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:29:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:29:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:29:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:29:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:29:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:29:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:29:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:29:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:29:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:29:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:29:47 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:29:47 lola-7 kernel: Call Trace: Aug 5 12:29:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:29:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:29:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:29:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:29:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:29:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:29:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:29:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:29:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:29:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:29:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:29:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:29:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:29:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:29:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:29:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:29:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:29:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:29:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 406 Aug 5 12:30:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 406 Aug 5 12:30:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 407 Aug 5 12:30:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 407 Aug 5 12:30:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 407 Aug 5 12:31:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:31:11 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:31:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:31:11 lola-7 kernel: CPU 27 Aug 5 12:31:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:31:11 lola-7 kernel: Aug 5 12:31:11 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:31:11 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:31:11 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:31:11 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:31:11 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:31:11 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:31:11 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:31:11 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:31:11 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:31:11 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:31:11 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:31:11 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:31:11 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:31:11 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:31:11 lola-7 kernel: Stack: Aug 5 12:31:11 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:31:11 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:31:11 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:31:11 lola-7 kernel: Call Trace: Aug 5 12:31:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:31:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:31:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:31:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:31:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:31:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:31:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:31:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:31:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:31:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:31:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:31:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:31:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:31:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:31:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:31:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:31:11 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:31:11 lola-7 kernel: Call Trace: Aug 5 12:31:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:31:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:31:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:31:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:31:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:31:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:31:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:31:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:31:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:31:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:31:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:31:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:31:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:31:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:31:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:31:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:31:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:31:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:31:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:31:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:31:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:32:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:32:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:32:35 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:32:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:32:35 lola-7 kernel: CPU 27 Aug 5 12:32:35 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:32:35 lola-7 kernel: Aug 5 12:32:35 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:32:35 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:32:35 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:32:35 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:32:35 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:32:35 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:32:35 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:32:35 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:32:35 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:32:35 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:32:35 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:32:35 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:32:35 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:32:35 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:32:35 lola-7 kernel: Stack: Aug 5 12:32:35 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:32:35 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:32:35 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:32:35 lola-7 kernel: Call Trace: Aug 5 12:32:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:32:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:32:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:32:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:32:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:32:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:32:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:32:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:32:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:32:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:32:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:32:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:32:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:32:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:32:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:32:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:32:35 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:32:35 lola-7 kernel: Call Trace: Aug 5 12:32:35 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:32:35 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:32:35 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:32:35 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:32:35 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:32:35 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:32:35 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:32:35 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:32:35 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:32:35 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:32:35 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:32:35 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:32:35 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:32:35 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:32:35 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:32:35 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:32:35 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:32:35 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:32:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:32:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:32:54 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1470425574/real 1470425574] req@ffff8806010176c0 x1541832706066400/t0(0) o38->soaked-MDT0001-lwp-OST0005@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1470425630 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 5 12:32:54 lola-7 kernel: Lustre: 17102:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Aug 5 12:33:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:33:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:33:35 lola-7 kernel: Lustre: 31666:0:(service.c:1335:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Aug 5 12:33:35 lola-7 kernel: req@ffff8801ff968850 x1541009530609072/t0(0) o4->34d7c76d-523f-2119-0f60-93198d84fad7@192.168.1.125@o2ib100:-1/-1 lens 608/0 e 0 to 0 dl 1470425620 ref 2 fl New:/2/ffffffff rc 0/-1 Aug 5 12:33:35 lola-7 kernel: Lustre: 31666:0:(service.c:1335:ptlrpc_at_send_early_reply()) Skipped 91 previous similar messages Aug 5 12:33:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 408 Aug 5 12:33:39 lola-7 kernel: LNet: Service thread pid 31502 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 5 12:33:39 lola-7 kernel: LNet: Skipped 4 previous similar messages Aug 5 12:33:39 lola-7 kernel: Pid: 31502, comm: ll_ost_io02_068 Aug 5 12:33:39 lola-7 kernel: Aug 5 12:33:39 lola-7 kernel: Call Trace: Aug 5 12:33:39 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:33:39 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:33:39 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:33:39 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:33:39 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:33:39 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:33:39 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:33:39 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ? pick_next_task_fair+0xd0/0x130 Aug 5 12:33:39 lola-7 kernel: [] ? schedule+0x176/0x3a0 Aug 5 12:33:39 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:33:39 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:33:39 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:33:39 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:33:39 lola-7 kernel: Aug 5 12:33:39 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425619.31502 Aug 5 12:33:39 lola-7 kernel: Pid: 31572, comm: ll_ost_io01_066 Aug 5 12:33:39 lola-7 kernel: Aug 5 12:33:39 lola-7 kernel: Call Trace: Aug 5 12:33:39 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:33:39 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:33:39 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:33:39 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:33:39 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 12:33:39 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:33:39 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:33:39 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:33:39 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:33:39 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:33:39 lola-7 kernel: Aug 5 12:33:39 lola-7 kernel: Pid: 31344, comm: ll_ost_io03_073 Aug 5 12:33:39 lola-7 kernel: Aug 5 12:33:39 lola-7 kernel: Call Trace: Aug 5 12:33:39 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:33:39 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:33:39 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:33:39 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:33:39 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ? check_preempt_curr+0x7c/0x90 Aug 5 12:33:39 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:33:39 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:33:39 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:33:39 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:33:39 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:33:39 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:33:39 lola-7 kernel: Aug 5 12:33:39 lola-7 kernel: Pid: 30816, comm: ll_ost_io03_034 Aug 5 12:33:39 lola-7 kernel: Aug 5 12:33:39 lola-7 kernel: Call Trace: Aug 5 12:33:39 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:33:39 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:33:39 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:33:39 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:33:39 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:33:39 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:33:39 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:33:39 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] ? do_dbs_timer+0x0/0x3bc [cpufreq_ondemand] Aug 5 12:33:39 lola-7 kernel: [] ? do_dbs_timer+0x0/0x3bc [cpufreq_ondemand] Aug 5 12:33:39 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:33:39 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:33:39 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:33:39 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:33:39 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:33:39 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:33:39 lola-7 kernel: Aug 5 12:33:39 lola-7 kernel: Pid: 30857, comm: ll_ost_io03_044 Aug 5 12:33:39 lola-7 kernel: Aug 5 12:33:39 lola-7 kernel: Call Trace: Aug 5 12:33:39 lola-7 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Aug 5 12:33:39 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:33:39 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:33:39 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:33:39 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:33:39 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:33:39 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:33:39 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:33:39 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:33:39 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:33:40 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:33:40 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:33:40 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:33:40 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:33:40 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:33:40 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:33:40 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:33:40 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:33:40 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:33:40 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:33:40 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:33:40 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:33:40 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:33:40 lola-7 kernel: Aug 5 12:33:40 lola-7 kernel: LNet: Service thread pid 31401 was inactive for 1200.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 5 12:33:41 lola-7 kernel: Lustre: soaked-OST0017: Client 9cff6698-27dd-142a-724f-49bd7985bea9 (at 192.168.1.127@o2ib100) reconnecting Aug 5 12:33:41 lola-7 kernel: Lustre: Skipped 16 previous similar messages Aug 5 12:33:41 lola-7 kernel: Lustre: soaked-OST0017: Connection restored to 9cff6698-27dd-142a-724f-49bd7985bea9 (at 192.168.1.127@o2ib100) Aug 5 12:33:41 lola-7 kernel: Lustre: Skipped 16 previous similar messages Aug 5 12:33:41 lola-7 kernel: LustreError: 31761:0:(service.c:2065:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-192.168.1.120@o2ib100: deadline 600:1s ago Aug 5 12:33:41 lola-7 kernel: req@ffff8803cbaba850 x1541009394952672/t0(0) o4->d3d6a29a-0595-e79b-060a-a9c1839ba326@192.168.1.120@o2ib100:-1/-1 lens 608/0 e 0 to 0 dl 1470425620 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 5 12:33:41 lola-7 kernel: Lustre: 31762:0:(service.c:2103:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8803f9a7c450 x1541009394958304/t0(0) o4->d3d6a29a-0595-e79b-060a-a9c1839ba326@192.168.1.120@o2ib100:-1/-1 lens 608/0 e 0 to 0 dl 1470425620 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Aug 5 12:33:41 lola-7 kernel: Lustre: 31762:0:(service.c:2103:ptlrpc_server_handle_request()) Skipped 1 previous similar message Aug 5 12:33:41 lola-7 kernel: LustreError: 31761:0:(service.c:2065:ptlrpc_server_handle_request()) Skipped 76 previous similar messages Aug 5 12:33:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 409 Aug 5 12:33:51 lola-7 kernel: Lustre: soaked-OST0005: Received MDS connection from 192.168.1.108@o2ib10, removing former export from 192.168.1.109@o2ib10 Aug 5 12:33:51 lola-7 kernel: Lustre: Skipped 2 previous similar messages Aug 5 12:33:56 lola-7 kernel: Lustre: soaked-OST0017: Export ffff880655577000 already connecting from 192.168.1.108@o2ib10 Aug 5 12:33:59 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:33:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:33:59 lola-7 kernel: CPU 27 Aug 5 12:33:59 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:33:59 lola-7 kernel: Aug 5 12:33:59 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:33:59 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:33:59 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:33:59 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:33:59 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:33:59 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:33:59 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:33:59 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:33:59 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:33:59 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:33:59 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:33:59 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:33:59 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:33:59 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:33:59 lola-7 kernel: Stack: Aug 5 12:33:59 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:33:59 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:33:59 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:33:59 lola-7 kernel: Call Trace: Aug 5 12:33:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:33:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:33:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:33:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:33:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:33:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:33:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:33:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:33:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:33:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:33:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:33:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:33:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:33:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:33:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:33:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:33:59 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:33:59 lola-7 kernel: Call Trace: Aug 5 12:33:59 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:33:59 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:33:59 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:33:59 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:33:59 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:33:59 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:33:59 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:33:59 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:33:59 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:33:59 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:33:59 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:33:59 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:33:59 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:33:59 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:33:59 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:33:59 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:33:59 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:33:59 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:34:02 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425642.30837 Aug 5 12:34:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 410 Aug 5 12:34:09 lola-7 kernel: LustreError: 167-0: soaked-MDT0001-lwp-OST0005: This client was evicted by soaked-MDT0001; in progress operations using this service will fail. Aug 5 12:34:11 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425651.31374 Aug 5 12:34:13 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425653.31570 Aug 5 12:34:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 411 Aug 5 12:34:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 412 Aug 5 12:34:46 lola-7 kernel: Lustre: soaked-OST0017: Export ffff880655577000 already connecting from 192.168.1.108@o2ib10 Aug 5 12:34:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 413 Aug 5 12:34:52 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425692.30475 Aug 5 12:34:54 lola-7 kernel: Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:79050 to 0x440000402:79713 Aug 5 12:34:54 lola-7 kernel: Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000400:80009 to 0x5c0000400:80961 Aug 5 12:34:54 lola-7 kernel: Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:79673 to 0x740000400:81121 Aug 5 12:35:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 413 Aug 5 12:35:20 lola-7 kernel: Lustre: soaked-OST0017: Export ffff880655577000 already connecting from 192.168.1.109@o2ib10 Aug 5 12:35:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 414 Aug 5 12:35:23 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:35:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:35:23 lola-7 kernel: CPU 27 Aug 5 12:35:23 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:35:23 lola-7 kernel: Aug 5 12:35:23 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:35:23 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:35:23 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:35:23 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:35:23 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:35:23 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:35:23 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:35:23 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:35:23 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:35:23 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:35:23 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:35:23 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:35:23 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:35:23 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:35:23 lola-7 kernel: Stack: Aug 5 12:35:23 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:35:23 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:35:23 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:35:23 lola-7 kernel: Call Trace: Aug 5 12:35:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:35:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:35:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:35:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:35:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:35:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:35:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:35:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:35:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:35:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:35:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:35:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:35:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:35:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:35:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:35:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:35:23 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:35:23 lola-7 kernel: Call Trace: Aug 5 12:35:23 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:35:23 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:35:23 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:35:23 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:35:23 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:35:23 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:35:23 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:35:23 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:35:23 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:35:23 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:35:23 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:35:23 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:35:23 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:35:23 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:35:23 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:35:23 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:35:23 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:35:23 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:35:24 lola-7 kernel: LustreError: 11-0: soaked-MDT0001-lwp-OST0011: operation obd_ping to node 192.168.1.108@o2ib10 failed: rc = -107 Aug 5 12:35:24 lola-7 kernel: Lustre: soaked-MDT0001-lwp-OST000b: Connection to soaked-MDT0001 (at 192.168.1.108@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Aug 5 12:35:24 lola-7 kernel: Lustre: Skipped 2 previous similar messages Aug 5 12:35:24 lola-7 kernel: LustreError: Skipped 2 previous similar messages Aug 5 12:35:28 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425728.31601 Aug 5 12:35:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 416 Aug 5 12:35:49 lola-7 kernel: LustreError: 167-0: soaked-MDT0001-lwp-OST000b: This client was evicted by soaked-MDT0001; in progress operations using this service will fail. Aug 5 12:35:49 lola-7 kernel: LustreError: Skipped 3 previous similar messages Aug 5 12:35:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 417 Aug 5 12:36:02 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425762.31610 Aug 5 12:36:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 418 Aug 5 12:36:10 lola-7 kernel: Lustre: soaked-OST0017: Export ffff880655577000 already connecting from 192.168.1.109@o2ib10 Aug 5 12:36:10 lola-7 kernel: Lustre: Skipped 1 previous similar message Aug 5 12:36:12 lola-7 kernel: Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:79050 to 0x440000402:79745 Aug 5 12:36:12 lola-7 kernel: Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:79673 to 0x740000400:81153 Aug 5 12:36:12 lola-7 kernel: Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000400:80009 to 0x5c0000400:80993 Aug 5 12:36:14 lola-7 kernel: LustreError: 167-0: soaked-MDT0001-lwp-OST0005: This client was evicted by soaked-MDT0001; in progress operations using this service will fail. Aug 5 12:36:14 lola-7 kernel: LustreError: Skipped 1 previous similar message Aug 5 12:36:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 419 Aug 5 12:36:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 419 Aug 5 12:36:47 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:36:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:36:47 lola-7 kernel: CPU 27 Aug 5 12:36:47 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:36:47 lola-7 kernel: Aug 5 12:36:47 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:36:47 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0x9/0x20 Aug 5 12:36:47 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:36:47 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:36:47 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:36:47 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:36:47 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:36:47 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:36:47 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:36:47 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:36:47 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:36:47 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:36:47 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:36:47 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:36:47 lola-7 kernel: Stack: Aug 5 12:36:47 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:36:47 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:36:47 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:36:47 lola-7 kernel: Call Trace: Aug 5 12:36:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:36:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:36:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:36:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:36:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:36:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:36:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:36:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:36:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:36:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:36:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:36:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:36:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:36:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:36:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:36:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:36:47 lola-7 kernel: Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff Aug 5 12:36:47 lola-7 kernel: Call Trace: Aug 5 12:36:47 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:36:47 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:36:47 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:36:47 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:36:47 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:36:47 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:36:47 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:36:47 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:36:47 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:36:47 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:36:47 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:36:47 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:36:47 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:36:47 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:36:47 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:36:47 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:36:47 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:36:47 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:36:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 420 Aug 5 12:37:00 lola-7 kernel: Lustre: soaked-OST0017: Export ffff880655577000 already connecting from 192.168.1.109@o2ib10 Aug 5 12:37:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 420 Aug 5 12:37:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 421 Aug 5 12:37:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 421 Aug 5 12:37:50 lola-7 kernel: Lustre: soaked-OST0017: Export ffff880655577000 already connecting from 192.168.1.109@o2ib10 Aug 5 12:37:50 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 421 Aug 5 12:38:05 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 421 Aug 5 12:38:11 lola-7 kernel: BUG: soft lockup - CPU#27 stuck for 67s! [ll_ost_io02_012:30458] Aug 5 12:38:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:38:11 lola-7 kernel: CPU 27 Aug 5 12:38:11 lola-7 kernel: Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm raid1 dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core joydev i2c_i801 ioatdma ses enclosure sg igb dca i2c_algo_bit i2c_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Aug 5 12:38:11 lola-7 kernel: Aug 5 12:38:11 lola-7 kernel: Pid: 30458, comm: ll_ost_io02_012 Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Aug 5 12:38:11 lola-7 kernel: RIP: 0010:[] [] __write_lock_failed+0xf/0x20 Aug 5 12:38:11 lola-7 kernel: RSP: 0018:ffff8803a57cf778 EFLAGS: 00000287 Aug 5 12:38:11 lola-7 kernel: RAX: ffff8806312f2440 RBX: ffff8803a57cf780 RCX: 00000000000279cf Aug 5 12:38:11 lola-7 kernel: RDX: ffff8807fd229200 RSI: ffff8807fd229200 RDI: ffff8806f5eb9f00 Aug 5 12:38:11 lola-7 kernel: RBP: ffffffff8100bc0e R08: ffff880678580b40 R09: 0000000000000008 Aug 5 12:38:11 lola-7 kernel: R10: 0000000b0004f0dc R11: 0000000000000000 R12: ffff8803a57cf6f0 Aug 5 12:38:11 lola-7 kernel: R13: 0000000000000000 R14: ffffffffa0944036 R15: ffff880624335320 Aug 5 12:38:11 lola-7 kernel: FS: 0000000000000000(0000) GS:ffff88044e560000(0000) knlGS:0000000000000000 Aug 5 12:38:11 lola-7 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 5 12:38:11 lola-7 kernel: CR2: 00000000004400c0 CR3: 0000000001a8d000 CR4: 00000000000407e0 Aug 5 12:38:11 lola-7 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 5 12:38:11 lola-7 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 5 12:38:11 lola-7 kernel: Process ll_ost_io02_012 (pid: 30458, threadinfo ffff8803a57cc000, task ffff8803a8c30040) Aug 5 12:38:11 lola-7 kernel: Stack: Aug 5 12:38:11 lola-7 kernel: ffffffff8153d007 ffff8803a57cf7b0 ffffffffa0fe3ab0 ffff880678580b50 Aug 5 12:38:11 lola-7 kernel: ffff8807fd229200 ffff8806f5eb9e48 ffff8803a57cf8d0 ffff8803a57cf7e0 Aug 5 12:38:11 lola-7 kernel: ffffffffa0fef79b 0000000000000000 ffff8807eedd85c0 ffff8803a57cf7e0 Aug 5 12:38:11 lola-7 kernel: Call Trace: Aug 5 12:38:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:38:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:38:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:38:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:38:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:38:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:38:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:38:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:38:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:38:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:38:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:38:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:38:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:38:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:38:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:38:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:38:11 lola-7 kernel: Code: 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 81 3f 00 00 00 01 <75> f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff c3 90 f0 ff 07 f3 Aug 5 12:38:11 lola-7 kernel: Call Trace: Aug 5 12:38:11 lola-7 kernel: [] ? _write_lock+0x17/0x20 Aug 5 12:38:11 lola-7 kernel: [] ? osd_object_sa_dirty_add+0x40/0x80 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? __osd_sa_xattr_schedule_update+0x4b/0x160 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? __osd_sa_xattr_set+0x25f/0x350 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? osd_xattr_set+0x2ec/0x4d0 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? dt_version_set+0x72/0x230 [obdclass] Aug 5 12:38:11 lola-7 kernel: [] ? tgt_txn_stop_cb+0x2c3/0x1160 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? dt_txn_hook_stop+0x5e/0x90 [obdclass] Aug 5 12:38:11 lola-7 kernel: [] ? osd_trans_stop+0x29c/0x520 [osd_zfs] Aug 5 12:38:11 lola-7 kernel: [] ? ofd_trans_stop+0x1f/0x60 [ofd] Aug 5 12:38:11 lola-7 kernel: [] ? ofd_commitrw_write+0x4d0/0xfa0 [ofd] Aug 5 12:38:11 lola-7 kernel: [] ? ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:38:11 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:38:11 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:38:11 lola-7 kernel: [] ? obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? enqueue_task+0x66/0x80 Aug 5 12:38:11 lola-7 kernel: [] ? check_preempt_curr+0x6d/0x90 Aug 5 12:38:11 lola-7 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Aug 5 12:38:11 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:38:11 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:38:11 lola-7 kernel: [] ? kthread+0x9e/0xc0 Aug 5 12:38:11 lola-7 kernel: [] ? child_rip+0xa/0x20 Aug 5 12:38:11 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:38:11 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:38:20 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 421 Aug 5 12:38:35 lola-7 sendmail[10599]: rejecting connections on daemon MTA: load average: 421 Aug 5 12:38:40 lola-7 kernel: Lustre: soaked-OST0017: Export ffff880655577000 already connecting from 192.168.1.109@o2ib10 Aug 5 12:38:48 lola-7 kernel: LNet: Service thread pid 31631 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 5 12:38:48 lola-7 kernel: LNet: Skipped 4 previous similar messages Aug 5 12:38:48 lola-7 kernel: Pid: 31631, comm: ll_ost_io01_071 Aug 5 12:38:48 lola-7 kernel: Aug 5 12:38:48 lola-7 kernel: Call Trace: Aug 5 12:38:48 lola-7 kernel: [] cv_wait_common+0x11d/0x130 [spl] Aug 5 12:38:48 lola-7 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 5 12:38:48 lola-7 kernel: [] __cv_wait+0x15/0x20 [spl] Aug 5 12:38:48 lola-7 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Aug 5 12:38:48 lola-7 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Aug 5 12:38:48 lola-7 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Aug 5 12:38:48 lola-7 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Aug 5 12:38:48 lola-7 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Aug 5 12:38:48 lola-7 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Aug 5 12:38:48 lola-7 kernel: [] ofd_commitrw_write+0x583/0xfa0 [ofd] Aug 5 12:38:48 lola-7 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Aug 5 12:38:48 lola-7 kernel: [] ? kzfree+0x31/0x40 Aug 5 12:38:48 lola-7 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Aug 5 12:38:48 lola-7 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Aug 5 12:38:48 lola-7 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Aug 5 12:38:48 lola-7 kernel: [] ? mod_timer+0x144/0x220 Aug 5 12:38:48 lola-7 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Aug 5 12:38:48 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:38:48 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:38:48 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:38:48 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:38:48 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:38:48 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:38:48 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:38:48 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:38:48 lola-7 kernel: Aug 5 12:38:48 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425928.31631 Aug 5 12:38:49 lola-7 kernel: Pid: 30418, comm: ll_ost03_016 Aug 5 12:38:49 lola-7 kernel: Aug 5 12:38:49 lola-7 kernel: Call Trace: Aug 5 12:38:49 lola-7 kernel: [] ? select_idle_sibling+0x95/0x150 Aug 5 12:38:49 lola-7 kernel: [] __mutex_lock_slowpath+0x96/0x210 Aug 5 12:38:49 lola-7 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Aug 5 12:38:49 lola-7 kernel: [] mutex_lock+0x2b/0x50 Aug 5 12:38:49 lola-7 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Aug 5 12:38:49 lola-7 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 5 12:38:49 lola-7 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Aug 5 12:38:49 lola-7 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Aug 5 12:38:49 lola-7 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 5 12:38:49 lola-7 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 5 12:38:49 lola-7 kernel: [] ? thread_return+0x4e/0x7d0 Aug 5 12:38:49 lola-7 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 5 12:38:49 lola-7 kernel: [] kthread+0x9e/0xc0 Aug 5 12:38:49 lola-7 kernel: [] child_rip+0xa/0x20 Aug 5 12:38:49 lola-7 kernel: [] ? kthread+0x0/0xc0 Aug 5 12:38:49 lola-7 kernel: [] ? child_rip+0x0/0x20 Aug 5 12:38:49 lola-7 kernel: Aug 5 12:38:49 lola-7 kernel: LustreError: dumping log to /tmp/lustre-log.1470425929.30418