Aug 25 10:35:51 feral28.ccs.ornl.gov kernel: [1448411.307417] Lustre: sultan-MDT000b: Connection restored to 36cf4012-aa44-00cd-a9bc-3d33ffda4df0 (at 10.37.202.11@o2ib1) Aug 25 18:35:55 feral28.ccs.ornl.gov kernel: [1477216.698647] Lustre: 21293:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1472164548/real 0] req@ffff880bbf0783c0 x1542128499454560/t0(0) o13->sultan-OST001f-osc-MDT000b@10.37.248.101@o2ib1:7/4 lens 224/368 e 0 to 1 dl 1472164555 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 25 18:35:55 feral28.ccs.ornl.gov kernel: [1477216.713634] Lustre: sultan-OST0007-osc-MDT000b: Connection to sultan-OST0007 (at 10.37.248.101@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 25 18:35:55 feral28.ccs.ornl.gov kernel: [1477216.713638] Lustre: Skipped 55 previous similar messages Aug 25 18:35:55 feral28.ccs.ornl.gov kernel: [1477216.752002] Lustre: 21293:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 213 previous similar messages Aug 25 18:35:55 feral28.ccs.ornl.gov kernel: [1477217.511716] Lustre: sultan-OST002f-osc-MDT000b: Connection to sultan-OST002f (at 10.37.248.101@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 25 18:35:55 feral28.ccs.ornl.gov kernel: [1477217.529674] Lustre: Skipped 6 previous similar messages Aug 25 18:35:56 feral28.ccs.ornl.gov kernel: [1477218.568782] Lustre: sultan-OST0027-osc-MDT000b: Connection to sultan-OST0027 (at 10.37.248.101@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 25 18:35:56 feral28.ccs.ornl.gov kernel: [1477218.586818] Lustre: Skipped 2 previous similar messages Aug 25 18:36:04 feral28.ccs.ornl.gov kernel: [1477226.458394] Lustre: sultan-MDT0006-osp-MDT000b: Connection to sultan-MDT0006 (at 10.37.248.161@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 25 18:36:04 feral28.ccs.ornl.gov kernel: [1477226.476347] Lustre: Skipped 31 previous similar messages Aug 25 18:36:04 feral28.ccs.ornl.gov kernel: [1477226.572406] LustreError: 166-1: MGC10.37.248.67@o2ib1: Connection to MGS (at 10.37.248.67@o2ib1) was lost; in progress operations using this service will fail Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.606551] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609550] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609552] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609613] CPU 5 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609614] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609652] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609654] Pid: 17158, comm: mdt01_007 Tainted: P -- ------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609657] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609668] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609669] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609671] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609673] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609674] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609676] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609678] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609680] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609681] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609683] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609685] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609687] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609689] Stack: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609690] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609692] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609694] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609697] Call Trace: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609756] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609804] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609856] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609903] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609949] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609973] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.609988] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610003] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610017] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610033] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610051] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610089] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610104] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610119] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610169] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610176] [] ? strlcpy+0x4a/0x60 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610193] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610207] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610222] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610237] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610281] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610322] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610379] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610433] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610487] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610539] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610543] [] ? thread_return+0x4e/0x7d0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610595] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610603] [] ? kthread+0x9e/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610608] [] ? child_rip+0xa/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610611] [] ? kthread+0x0/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610613] [] ? child_rip+0x0/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610614] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610630] Call Trace: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610673] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610719] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610769] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610815] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610862] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610877] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610888] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610900] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610910] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610925] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610941] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610970] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610985] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.610999] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611049] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611053] [] ? strlcpy+0x4a/0x60 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611070] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611084] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611099] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611114] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611158] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611198] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611252] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611306] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611360] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611412] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611415] [] ? thread_return+0x4e/0x7d0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611467] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611470] [] ? kthread+0x9e/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611473] [] ? child_rip+0xa/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611476] [] ? kthread+0x0/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.611478] [] ? child_rip+0x0/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613550] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613552] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613597] CPU 6 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613598] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613635] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613637] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613640] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613645] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613647] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613648] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613650] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613652] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613653] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613655] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613657] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613659] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613660] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613662] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613664] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613665] Stack: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613666] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613669] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613671] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613674] Call Trace: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613712] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613759] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613809] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613856] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613902] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613919] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613930] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613945] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.613958] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614009] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614020] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614035] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614068] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614084] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614113] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614128] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614143] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614193] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614198] [] ? strlcpy+0x4a/0x60 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614214] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614229] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614243] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614258] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614302] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614342] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614397] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614451] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614504] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614556] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614560] [] ? thread_return+0x4e/0x7d0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614611] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614615] [] ? kthread+0x9e/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614619] [] ? child_rip+0xa/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614621] [] ? kthread+0x0/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614623] [] ? child_rip+0x0/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614625] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614640] Call Trace: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614683] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614729] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614779] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614825] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614872] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614888] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614898] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614910] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614922] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614972] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614983] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.614998] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615027] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615043] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615071] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615086] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615101] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615151] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615155] [] ? strlcpy+0x4a/0x60 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615171] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615186] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615201] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615215] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615259] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615300] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615354] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615407] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615461] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615513] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615516] [] ? thread_return+0x4e/0x7d0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615567] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615571] [] ? kthread+0x9e/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615574] [] ? child_rip+0xa/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615576] [] ? kthread+0x0/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.615579] [] ? child_rip+0x0/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616550] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616551] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616594] CPU 7 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616595] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616631] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616633] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616636] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616641] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616643] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616644] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616646] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616648] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616649] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616651] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616654] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616655] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616657] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616659] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616661] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616662] Stack: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616663] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616665] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616667] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616670] Call Trace: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616704] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616751] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616801] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616847] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616894] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616910] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616920] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616933] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616943] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616959] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.616975] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617004] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617019] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617034] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617084] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617088] [] ? strlcpy+0x4a/0x60 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617104] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617119] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617134] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617149] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617193] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617233] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617287] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617341] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617395] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617447] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617450] [] ? thread_return+0x4e/0x7d0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617501] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617505] [] ? kthread+0x9e/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617508] [] ? child_rip+0xa/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617511] [] ? kthread+0x0/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617513] [] ? child_rip+0x0/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617514] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617529] Call Trace: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617572] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617618] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617668] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617715] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617761] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617777] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617787] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617799] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617810] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617825] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617841] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617869] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617884] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617899] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617949] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617953] [] ? strlcpy+0x4a/0x60 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617969] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617984] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.617999] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618014] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618057] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618098] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618152] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618206] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618259] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618311] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618314] [] ? thread_return+0x4e/0x7d0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618366] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618369] [] ? kthread+0x9e/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618372] [] ? child_rip+0xa/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618375] [] ? kthread+0x0/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477293.618377] [] ? child_rip+0x0/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.136079] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.233435] CPU 4 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.235523] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.333286] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.335141] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.348602] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.357294] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.363162] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.371110] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.379040] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.386971] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.394883] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.402797] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.411644] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.417964] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.425825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.433703] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.441625] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.451175] Stack: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.453559] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.461445] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.469991] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.478849] Call Trace: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.481753] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.489095] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.497081] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.504721] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.512191] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.519711] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.527315] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.534676] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.541824] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.548736] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.556546] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.563761] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.572102] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.578472] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.585466] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.593438] [] ? strlcpy+0x4a/0x60 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.599076] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.605990] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.613391] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.620533] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.627877] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.635483] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.643522] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.650855] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.657770] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.665595] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.672636] [] ? thread_return+0x4e/0x7d0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.678934] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.685834] [] ? kthread+0x9e/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.691452] [] ? child_rip+0xa/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.697172] [] ? kthread+0x0/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.702648] [] ? child_rip+0x0/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.708386] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.729800] Call Trace: Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.732729] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.740152] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.748223] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.755847] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.763223] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.770784] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.778422] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.785801] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.792888] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.799835] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.807697] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.814878] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.823158] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.829520] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.836541] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.844511] [] ? strlcpy+0x4a/0x60 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.850147] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.856975] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.864361] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.871528] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.881516] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.889061] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.897135] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.904580] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.911496] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.919399] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.926413] [] ? thread_return+0x4e/0x7d0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.932720] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.939648] [] ? kthread+0x9e/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.945264] [] ? child_rip+0xa/0x20 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.951027] [] ? kthread+0x0/0xc0 Aug 25 18:37:15 feral28.ccs.ornl.gov kernel: [1477296.956487] [] ? child_rip+0x0/0x20 Aug 25 18:37:37 feral28.ccs.ornl.gov kernel: [1477319.623708] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Aug 25 18:37:37 feral28.ccs.ornl.gov kernel: [1477319.634809] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.156@o2ib1 (134): c: 60, oc: 0, rc: 63 Aug 25 18:37:38 feral28.ccs.ornl.gov kernel: [1477320.623791] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 18:37:38 feral28.ccs.ornl.gov kernel: [1477320.634818] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Aug 25 18:37:38 feral28.ccs.ornl.gov kernel: [1477320.645776] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.161@o2ib1 (138): c: 61, oc: 0, rc: 63 Aug 25 18:37:38 feral28.ccs.ornl.gov kernel: [1477320.659204] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 2 previous similar messages Aug 25 18:37:40 feral28.ccs.ornl.gov kernel: [1477322.623952] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 3 seconds Aug 25 18:37:40 feral28.ccs.ornl.gov kernel: [1477322.634973] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Aug 25 18:37:40 feral28.ccs.ornl.gov kernel: [1477322.645985] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.67@o2ib1 (140): c: 61, oc: 0, rc: 63 Aug 25 18:37:40 feral28.ccs.ornl.gov kernel: [1477322.659336] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 2 previous similar messages Aug 25 18:37:44 feral28.ccs.ornl.gov kernel: [1477326.624298] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 6 seconds Aug 25 18:37:44 feral28.ccs.ornl.gov kernel: [1477326.635335] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Aug 25 18:37:44 feral28.ccs.ornl.gov kernel: [1477326.646312] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.83@o2ib1 (118): c: 35, oc: 0, rc: 63 Aug 25 18:37:44 feral28.ccs.ornl.gov kernel: [1477326.659664] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 2 previous similar messages Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.613168] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616168] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616170] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616212] CPU 5 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616212] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616248] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616250] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616253] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616257] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616259] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616260] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616262] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616263] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616265] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616267] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616269] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616270] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616272] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616276] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616277] Stack: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616278] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616280] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616283] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616285] Call Trace: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616329] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616376] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616426] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616472] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616519] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616535] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616546] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616558] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616568] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616583] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616600] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616629] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616643] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616658] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616709] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616713] [] ? strlcpy+0x4a/0x60 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616729] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616744] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616759] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616774] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616817] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616858] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616912] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.616966] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617019] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617071] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617075] [] ? thread_return+0x4e/0x7d0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617126] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617130] [] ? kthread+0x9e/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617133] [] ? child_rip+0xa/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617135] [] ? kthread+0x0/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617137] [] ? child_rip+0x0/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617139] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617154] Call Trace: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617197] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617243] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617293] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617339] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617386] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617401] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617412] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617424] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617434] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617449] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617465] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617495] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617509] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617524] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617574] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617578] [] ? strlcpy+0x4a/0x60 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617594] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617609] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617623] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617638] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617682] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617722] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617776] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617830] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617884] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617935] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617939] [] ? thread_return+0x4e/0x7d0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617991] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617994] [] ? kthread+0x9e/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.617997] [] ? child_rip+0xa/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.618000] [] ? kthread+0x0/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.618002] [] ? child_rip+0x0/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620168] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620169] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620212] CPU 6 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620213] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620250] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620251] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620255] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620259] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620261] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620263] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620265] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620266] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620268] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620270] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620272] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620274] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620276] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620277] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620279] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620281] Stack: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620282] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620284] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620286] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620288] Call Trace: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620323] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620370] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620420] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620466] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620513] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620529] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620539] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620551] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620563] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620614] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620624] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620640] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620669] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620686] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620714] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620729] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620744] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620794] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620798] [] ? strlcpy+0x4a/0x60 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620814] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620829] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620844] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620859] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620903] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620943] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.620997] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621051] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621105] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621156] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621160] [] ? thread_return+0x4e/0x7d0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621212] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621215] [] ? kthread+0x9e/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621218] [] ? child_rip+0xa/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621221] [] ? kthread+0x0/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621223] [] ? child_rip+0x0/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621224] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621240] Call Trace: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621282] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621329] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621378] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621425] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621471] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621487] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621498] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621509] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621521] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621572] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621582] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621598] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621627] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621643] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621671] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621686] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621701] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621752] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621755] [] ? strlcpy+0x4a/0x60 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621772] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621786] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621801] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621816] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621860] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621900] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.621954] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.622008] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.622061] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.622113] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.622116] [] ? thread_return+0x4e/0x7d0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.622168] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.622171] [] ? kthread+0x9e/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.622174] [] ? child_rip+0xa/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.622177] [] ? kthread+0x0/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.622179] [] ? child_rip+0x0/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623167] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623169] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623209] CPU 7 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623210] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623247] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623249] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623252] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623256] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623257] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623259] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623261] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623262] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623264] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623266] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623268] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623269] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623271] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623273] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623275] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623276] Stack: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623277] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623279] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623282] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623284] Call Trace: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623318] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623364] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623414] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623461] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623508] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623523] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623534] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623546] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623556] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623571] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623587] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623616] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623631] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623646] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623696] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623700] [] ? strlcpy+0x4a/0x60 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623717] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623731] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623746] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623761] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623805] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623846] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623900] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.623954] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624007] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624059] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624062] [] ? thread_return+0x4e/0x7d0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624114] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624117] [] ? kthread+0x9e/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624120] [] ? child_rip+0xa/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624123] [] ? kthread+0x0/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624125] [] ? child_rip+0x0/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624126] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624142] Call Trace: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624184] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624231] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624280] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624327] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624374] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624389] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624400] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624412] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624422] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624437] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624453] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624482] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624497] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624512] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624562] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624566] [] ? strlcpy+0x4a/0x60 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624582] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624597] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624612] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624627] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624671] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624711] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624765] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624819] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624872] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624924] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624928] [] ? thread_return+0x4e/0x7d0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624979] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624983] [] ? kthread+0x9e/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624985] [] ? child_rip+0xa/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624988] [] ? kthread+0x0/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477377.624990] [] ? child_rip+0x0/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.139403] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.236722] CPU 4 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.238811] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.336384] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.338250] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.351761] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.360428] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.366365] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.374364] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.382293] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.390182] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.398103] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.405981] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.414969] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.421322] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.429192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.437105] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.445027] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.454464] Stack: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.456822] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.464717] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.473245] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.484677] Call Trace: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.487562] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.494913] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.503030] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.510697] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.518220] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.525791] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.533395] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.540780] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.547850] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.554764] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.562633] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.569856] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.578164] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.584535] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.591599] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.599466] [] ? strlcpy+0x4a/0x60 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.605059] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.611844] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.619194] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.626370] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.633654] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.641216] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.649273] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.656761] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.663738] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.671613] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.678656] [] ? thread_return+0x4e/0x7d0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.684961] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.691864] [] ? kthread+0x9e/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.697444] [] ? child_rip+0xa/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.703120] [] ? kthread+0x0/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.708624] [] ? child_rip+0x0/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.714300] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.735759] Call Trace: Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.738679] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.746058] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.754104] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.761788] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.769260] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.776821] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.784416] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.791749] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.798839] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.805740] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.813621] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.820740] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.829012] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.835383] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.842421] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.850331] [] ? strlcpy+0x4a/0x60 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.855951] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.862805] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.870120] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.877305] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.884545] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.892073] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.900112] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.907514] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.914507] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.922341] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.929443] [] ? thread_return+0x4e/0x7d0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.935671] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.942556] [] ? kthread+0x9e/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.948172] [] ? child_rip+0xa/0x20 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.953883] [] ? kthread+0x0/0xc0 Aug 25 18:38:39 feral28.ccs.ornl.gov kernel: [1477380.959401] [] ? child_rip+0x0/0x20 Aug 25 18:38:58 feral28.ccs.ornl.gov kernel: [1477400.630180] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 18:38:58 feral28.ccs.ornl.gov kernel: [1477400.641297] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Aug 25 18:38:58 feral28.ccs.ornl.gov kernel: [1477400.652456] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.77@o2ib1 (193): c: 63, oc: 0, rc: 63 Aug 25 18:38:58 feral28.ccs.ornl.gov kernel: [1477400.665763] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 2 previous similar messages Aug 25 18:39:08 feral28.ccs.ornl.gov kernel: [1477410.630877] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 18:39:08 feral28.ccs.ornl.gov kernel: [1477410.641953] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 1 previous similar message Aug 25 18:39:08 feral28.ccs.ornl.gov kernel: [1477410.652912] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.119@o2ib1 (211): c: 62, oc: 0, rc: 63 Aug 25 18:39:08 feral28.ccs.ornl.gov kernel: [1477410.666342] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 1 previous similar message Aug 25 18:39:25 feral28.ccs.ornl.gov kernel: [1477427.324322] Lustre: sultan-MDT000b: Client 1f0727a3-d49c-044b-6def-11f5d586ab1d (at 26@gni1) reconnecting Aug 25 18:39:25 feral28.ccs.ornl.gov kernel: [1477427.334743] Lustre: sultan-MDT000b: Connection restored to c60f26de-f3d6-6cfc-b87c-120a7b1f0eed (at 26@gni1) Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.619786] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622786] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622788] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622830] CPU 5 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622831] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622868] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622870] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622872] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622877] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622878] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622880] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622882] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622883] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622885] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622887] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622889] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622891] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622893] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622894] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622896] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622898] Stack: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622898] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622900] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622903] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622905] Call Trace: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622949] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.622996] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623045] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623092] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623139] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623154] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623165] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623177] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623188] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623203] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623219] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623249] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623263] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623278] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623329] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623333] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623349] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623363] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623378] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623393] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623437] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623477] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623531] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623585] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623638] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623690] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623694] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623746] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623749] [] ? kthread+0x9e/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623752] [] ? child_rip+0xa/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623754] [] ? kthread+0x0/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623757] [] ? child_rip+0x0/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623758] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623773] Call Trace: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623816] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623863] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623912] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.623959] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624005] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624021] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624031] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624043] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624054] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624069] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624085] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624114] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624129] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624143] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624194] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624198] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624214] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624228] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624243] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624258] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624302] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624342] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624396] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624450] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624504] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624556] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624559] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624611] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624615] [] ? kthread+0x9e/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624617] [] ? child_rip+0xa/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624620] [] ? kthread+0x0/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.624622] [] ? child_rip+0x0/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626786] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626788] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626830] CPU 6 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626831] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626868] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626870] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626873] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626878] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626880] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626881] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626883] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626885] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626886] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626889] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626891] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626893] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626896] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626898] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626899] Stack: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626900] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626902] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626905] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626907] Call Trace: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626942] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.626989] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627039] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627085] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627132] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627147] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627158] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627170] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627182] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627232] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627243] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627259] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627290] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627306] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627335] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627350] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627365] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627415] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627419] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627435] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627450] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627465] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627480] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627524] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627564] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627618] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627672] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627726] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627778] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627781] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627833] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627837] [] ? kthread+0x9e/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627839] [] ? child_rip+0xa/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627842] [] ? kthread+0x0/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627844] [] ? child_rip+0x0/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627846] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627861] Call Trace: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627904] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.627950] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628000] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628046] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628093] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628109] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628119] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628131] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628143] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628193] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628204] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628219] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628249] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628265] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628294] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628309] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628323] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628374] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628378] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628394] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628408] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628423] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628438] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628482] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628522] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628576] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628630] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628684] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628736] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628739] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628791] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628795] [] ? kthread+0x9e/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628798] [] ? child_rip+0xa/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628800] [] ? kthread+0x0/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.628802] [] ? child_rip+0x0/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629785] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629787] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629828] CPU 7 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629828] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629864] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629866] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629869] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629873] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629875] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629876] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629878] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629879] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629881] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629883] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629885] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629886] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629888] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629890] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629892] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629893] Stack: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629894] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629896] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629898] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629901] Call Trace: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629935] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.629982] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630032] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630078] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630125] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630141] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630152] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630164] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630174] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630189] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630206] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630234] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630249] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630264] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630314] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630318] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630334] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630349] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630364] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630379] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630423] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630463] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630517] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630571] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630625] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630677] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630680] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630732] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630735] [] ? kthread+0x9e/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630738] [] ? child_rip+0xa/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630741] [] ? kthread+0x0/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630743] [] ? child_rip+0x0/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630744] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630760] Call Trace: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630802] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630849] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630899] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630945] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.630992] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631007] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631018] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631030] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631040] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631055] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631071] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631100] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631115] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631130] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631180] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631184] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631200] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631215] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631229] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631244] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631288] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631328] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631382] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631436] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631490] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631542] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631545] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631597] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631600] [] ? kthread+0x9e/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631603] [] ? child_rip+0xa/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631606] [] ? kthread+0x0/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477461.631608] [] ? child_rip+0x0/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.150058] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.247259] CPU 4 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.249364] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.347024] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.348906] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.362380] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.371006] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.376934] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.384820] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.392742] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.400690] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.408586] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.416508] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.425486] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.431847] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.439813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.447768] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.455655] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.465187] Stack: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.467581] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.475434] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.483919] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.492768] Call Trace: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.495671] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.503099] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.511146] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.518806] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.526285] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.534149] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.541804] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.549060] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.556141] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.563001] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.570819] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.578053] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.586229] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.592539] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.599481] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.607244] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.612916] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.619692] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.627059] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.634183] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.641516] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.649099] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.657181] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.664618] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.671567] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.679470] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.686527] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.692835] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.699772] [] ? kthread+0x9e/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.705379] [] ? child_rip+0xa/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.711046] [] ? kthread+0x0/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.716488] [] ? child_rip+0x0/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.722165] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.743736] Call Trace: Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.746622] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.754051] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.762091] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.769749] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.777177] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.784643] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.792195] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.799572] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.806590] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.813468] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.821286] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.828407] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.836686] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.842962] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.849940] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.857815] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.863435] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.870219] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.877543] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.884737] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.892037] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.899652] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.907734] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.915067] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.922035] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.929920] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.937040] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.943388] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.950265] [] ? kthread+0x9e/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.955862] [] ? child_rip+0xa/0x20 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.961549] [] ? kthread+0x0/0xc0 Aug 25 18:40:03 feral28.ccs.ornl.gov kernel: [1477464.967078] [] ? child_rip+0x0/0x20 Aug 25 18:40:46 feral28.ccs.ornl.gov kernel: [1477508.375401] Lustre: sultan-MDT000b: Client 1f0727a3-d49c-044b-6def-11f5d586ab1d (at 26@gni1) reconnecting Aug 25 18:40:46 feral28.ccs.ornl.gov kernel: [1477508.385875] Lustre: sultan-MDT000b: Connection restored to c60f26de-f3d6-6cfc-b87c-120a7b1f0eed (at 26@gni1) Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.017748] LNet: Service thread pid 21485 was inactive for 300.06s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.036054] Pid: 21485, comm: mdt01_000 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.040409] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.040410] Call Trace: Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.045147] [] ? __wake_up+0x53/0x70 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.051015] [] ? ptlrpc_check_set+0x1259/0x1d40 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.058694] [] ? apic_timer_interrupt+0xe/0x20 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.065460] [] ? _spin_lock+0x21/0x30 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.071418] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.078877] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.086965] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.094704] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.102181] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.109722] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.117282] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.124623] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.132564] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.139678] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.146751] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.153639] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.161528] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.168800] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.175937] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.184303] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.190604] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.197625] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.205537] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.213840] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.220686] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.228071] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.235195] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.242480] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.250043] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.258119] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.265449] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.272337] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.280165] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.287222] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.293552] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.300441] [] ? kthread+0x9e/0xc0 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.306026] [] ? child_rip+0xa/0x20 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.311714] [] ? kthread+0x0/0xc0 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.317242] [] ? child_rip+0x0/0x20 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.322943] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.324790] LustreError: dumping log to /tmp/lustre-log.1472164848.21485 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.362095] Pid: 25186, comm: mdt01_006 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.366412] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.366413] Call Trace: Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.371146] [] ? LNetMDUnlink+0xd4/0x160 [lnet] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.377984] [] ? at_measured+0x108/0x380 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.385044] [] ? lprocfs_stats_alloc_one+0x132/0x4d0 [obdclass] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.393427] [] __cond_resched+0x2a/0x40 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.399584] [] _cond_resched+0x30/0x40 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.405566] [] __kmalloc+0x138/0x230 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.411403] [] lprocfs_stats_alloc_one+0x132/0x4d0 [obdclass] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.419630] [] lprocfs_counter_add+0x1a8/0x1c0 [obdclass] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.427544] [] ldlm_pool_add+0x5b/0x130 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.434509] [] ldlm_grant_lock+0x15e/0x760 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.441771] [] ldlm_lock_enqueue+0x67a/0x990 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.449218] [] ldlm_cli_enqueue_fini+0x914/0xe30 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.457116] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.464846] [] ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.472114] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.479673] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.487325] [] osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.494460] [] lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.501382] [] mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.508096] [] mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.515658] [] mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.522662] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.530909] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.537234] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.544203] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.552107] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.557680] [] mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.564282] [] mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.571433] [] mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.578373] [] mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.585467] [] ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.592867] [] ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.600693] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.608108] [] tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.614879] [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.622434] [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.629296] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.635591] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.642476] [] kthread+0x9e/0xc0 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.647883] [] child_rip+0xa/0x20 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.653406] [] ? kthread+0x0/0xc0 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.658916] [] ? child_rip+0x0/0x20 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.664580] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.666477] LNet: Service thread pid 17159 was inactive for 300.71s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.684571] LNet: Skipped 1 previous similar message Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.690028] Pid: 17159, comm: mdt01_008 Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.694343] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.694344] Call Trace: Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.699070] [] ? ptl_send_rpc+0x685/0xea0 [ptlrpc] Aug 25 18:40:48 feral28.ccs.ornl.gov kernel: [1477510.706112] [] schedule_timeout+0x192/0x2e0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.712520] [] ? process_timeout+0x0/0x10 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.718844] [] ptlrpc_set_wait+0x321/0x960 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.726079] [] ? default_wake_function+0x0/0x20 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.732934] [] ? lustre_msg_set_jobid+0xf5/0x130 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.740784] [] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.748076] [] ldlm_cli_enqueue+0x37e/0x870 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.755320] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.762877] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.770498] [] osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.777673] [] lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.784553] [] mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.791255] [] mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.798836] [] mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.805862] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.814243] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.820638] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.827677] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.835583] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.841236] [] mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.847889] [] mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.855069] [] mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.862146] [] mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.869319] [] ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.876702] [] ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.884610] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.892081] [] tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.898895] [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.906546] [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.913433] [] ? pick_next_task_fair+0xd0/0x130 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.920251] [] ? schedule+0x176/0x3a0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.926183] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.933125] [] kthread+0x9e/0xc0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.938572] [] child_rip+0xa/0x20 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.944094] [] ? kthread+0x0/0xc0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.949633] [] ? child_rip+0x0/0x20 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.955283] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.957118] Pid: 17158, comm: mdt01_007 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.961427] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.961428] Call Trace: Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.966166] [] ? __wake_up+0x53/0x70 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.972031] [] ? ptlrpc_check_set+0x1259/0x1d40 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.979685] [] ? reschedule_interrupt+0xe/0x20 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.986379] [] ? _spin_lock+0x1e/0x30 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.992326] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477510.999729] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.007762] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.015398] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.022860] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.030366] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.038007] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.045347] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.052423] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.059315] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.067119] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.074331] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.082613] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.088984] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.095979] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.103863] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.109558] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.116438] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.123857] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.131016] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.138317] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.145967] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.154103] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.161565] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.168557] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.176470] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.183545] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.189842] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.196691] [] ? kthread+0x9e/0xc0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.202318] [] ? child_rip+0xa/0x20 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.208026] [] ? kthread+0x0/0xc0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.213476] [] ? child_rip+0x0/0x20 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.219162] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.221028] Pid: 21487, comm: mdt01_002 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.225360] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.225361] Call Trace: Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.230060] [] ? __wake_up+0x53/0x70 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.235922] [] ? ptlrpc_check_set+0x1259/0x1d40 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.243475] [] ? reschedule_interrupt+0xe/0x20 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.250158] [] ? _spin_lock+0x18/0x30 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.256064] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.263447] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.271483] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.279115] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.289265] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.296818] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.304402] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.311775] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.318826] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.325728] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.333565] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.340694] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.348991] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.355361] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.362375] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.370250] [] ? strlcpy+0x4a/0x60 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.375893] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.382738] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.390158] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.397377] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.404731] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.412339] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.420413] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.427831] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.434860] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.442769] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.449874] [] ? thread_return+0x4e/0x7d0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.456142] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.463133] [] ? kthread+0x9e/0xc0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.468721] [] ? child_rip+0xa/0x20 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.474424] [] ? kthread+0x0/0xc0 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.479902] [] ? child_rip+0x0/0x20 Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.485603] Aug 25 18:40:49 feral28.ccs.ornl.gov kernel: [1477511.487470] LNet: Service thread pid 21486 was inactive for 301.53s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Aug 25 18:40:58 feral28.ccs.ornl.gov kernel: [1477519.753019] Lustre: sultan-MDT000b: haven't heard from client sultan-MDT000e-mdtlov_UUID (at 10.37.248.169@o2ib1) in 339 seconds. I think it's dead, and I am evicting it. exp ffff8817ca669400, cur 1472164858 expire 1472164636 last 1472164519 Aug 25 18:40:58 feral28.ccs.ornl.gov kernel: [1477519.776003] Lustre: Skipped 21 previous similar messages Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.626405] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629405] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629406] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629449] CPU 5 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629450] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629487] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629488] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629491] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629496] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629497] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629499] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629500] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629502] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629504] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629506] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629508] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629509] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629513] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629515] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629516] Stack: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629517] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629519] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629522] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629524] Call Trace: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629568] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629615] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629664] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629711] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629758] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629774] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629785] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629797] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629807] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629823] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629839] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629868] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629883] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629898] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629948] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629952] [] ? strlcpy+0x4a/0x60 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629968] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629983] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.629998] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630013] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630056] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630097] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630151] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630204] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630258] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630310] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630313] [] ? thread_return+0x4e/0x7d0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630365] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630368] [] ? kthread+0x9e/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630371] [] ? child_rip+0xa/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630374] [] ? kthread+0x0/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630376] [] ? child_rip+0x0/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630377] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630393] Call Trace: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630435] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630482] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630532] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630578] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630625] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630641] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630652] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630663] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630674] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630689] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630705] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630734] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630748] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630763] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630814] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630818] [] ? strlcpy+0x4a/0x60 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630834] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630848] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630863] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630878] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630922] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.630962] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631016] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631070] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631124] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631175] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631179] [] ? thread_return+0x4e/0x7d0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631231] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631234] [] ? kthread+0x9e/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631237] [] ? child_rip+0xa/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631239] [] ? kthread+0x0/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.631242] [] ? child_rip+0x0/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633405] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633407] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633450] CPU 6 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633450] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633487] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633489] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633492] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633497] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633499] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633500] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633502] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633504] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633505] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633508] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633510] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633511] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633515] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633517] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633518] Stack: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633519] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633521] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633523] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633525] Call Trace: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633561] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633607] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633657] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633704] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633750] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633766] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633777] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633789] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633801] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633852] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633862] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633877] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633909] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633925] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633954] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633968] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.633983] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634034] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634038] [] ? strlcpy+0x4a/0x60 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634054] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634069] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634084] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634098] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634142] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634182] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634237] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634291] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634344] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634396] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634400] [] ? thread_return+0x4e/0x7d0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634452] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634455] [] ? kthread+0x9e/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634458] [] ? child_rip+0xa/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634461] [] ? kthread+0x0/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634463] [] ? child_rip+0x0/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634464] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634479] Call Trace: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634522] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634568] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634618] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634665] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634711] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634727] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634737] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634749] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634761] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634811] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634822] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634837] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634866] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634882] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634911] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634925] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634941] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634991] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.634994] [] ? strlcpy+0x4a/0x60 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635011] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635025] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635040] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635055] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635099] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635139] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635193] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635247] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635301] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635352] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635356] [] ? thread_return+0x4e/0x7d0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635407] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635411] [] ? kthread+0x9e/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635414] [] ? child_rip+0xa/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635416] [] ? kthread+0x0/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.635418] [] ? child_rip+0x0/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636404] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636406] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636447] CPU 7 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636447] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636484] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636485] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636488] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636492] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636494] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636495] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636497] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636498] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636500] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636502] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636504] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636505] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636507] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636509] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636511] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636512] Stack: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636513] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636515] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636517] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636519] Call Trace: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636553] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636600] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636649] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636696] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636743] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636758] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636769] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636781] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636791] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636806] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636822] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636851] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636866] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636881] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636931] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636935] [] ? strlcpy+0x4a/0x60 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636951] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636966] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636981] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.636996] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637040] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637080] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637134] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637188] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637242] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637293] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637297] [] ? thread_return+0x4e/0x7d0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637348] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637352] [] ? kthread+0x9e/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637355] [] ? child_rip+0xa/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637358] [] ? kthread+0x0/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637360] [] ? child_rip+0x0/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637361] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637376] Call Trace: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637419] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637465] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637515] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637561] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637608] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637623] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637634] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637646] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637656] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637671] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637687] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637716] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637731] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637746] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637796] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637800] [] ? strlcpy+0x4a/0x60 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637816] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637831] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637846] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637860] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637904] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637944] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.637998] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.638052] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.638106] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.638157] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.638161] [] ? thread_return+0x4e/0x7d0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.638213] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.638216] [] ? kthread+0x9e/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.638219] [] ? child_rip+0xa/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.638222] [] ? kthread+0x0/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477545.638224] [] ? child_rip+0x0/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477546.641595] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477546.641602] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 1 previous similar message Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477546.641613] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.156@o2ib1 (205): c: 63, oc: 0, rc: 63 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477546.641618] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 1 previous similar message Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.199239] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.296264] CPU 4 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.298398] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.396074] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.397965] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.411398] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.420100] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.425985] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.433897] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.441758] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.449645] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.457558] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.465481] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.474321] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.480708] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.488603] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.496525] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.504404] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.516537] Stack: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.518981] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.526851] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.535336] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.544115] Call Trace: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.547027] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.554414] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.562477] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.570101] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.577597] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.585116] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.592737] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.600054] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.607081] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.614037] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.621847] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.628983] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.637308] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.643661] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.650681] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.658600] [] ? strlcpy+0x4a/0x60 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.664255] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.671057] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.678458] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.685667] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.692944] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.700524] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.708562] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.715973] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.722976] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.730939] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.737989] [] ? thread_return+0x4e/0x7d0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.744284] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.751207] [] ? kthread+0x9e/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.756787] [] ? child_rip+0xa/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.762455] [] ? kthread+0x0/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.768010] [] ? child_rip+0x0/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.773746] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.795137] Call Trace: Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.798039] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.805409] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.813438] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.821175] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.828698] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.836224] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.843827] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.851127] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.858214] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.865127] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.872955] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.880108] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.888406] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.894749] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.901771] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.909674] [] ? strlcpy+0x4a/0x60 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.915319] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.922148] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.929521] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.936656] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.943922] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.951545] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.959628] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.967064] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.974056] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.981874] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.988967] [] ? thread_return+0x4e/0x7d0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477548.995239] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477549.002122] [] ? kthread+0x9e/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477549.007729] [] ? child_rip+0xa/0x20 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477549.013407] [] ? kthread+0x0/0xc0 Aug 25 18:41:27 feral28.ccs.ornl.gov kernel: [1477549.018928] [] ? child_rip+0x0/0x20 Aug 25 18:42:45 feral28.ccs.ornl.gov kernel: [1477627.647981] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 18:42:45 feral28.ccs.ornl.gov kernel: [1477627.659361] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 8 previous similar messages Aug 25 18:42:45 feral28.ccs.ornl.gov kernel: [1477627.670502] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.77@o2ib1 (222): c: 63, oc: 0, rc: 63 Aug 25 18:42:45 feral28.ccs.ornl.gov kernel: [1477627.683905] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 8 previous similar messages Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.633023] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636023] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636025] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636067] CPU 5 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636068] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636104] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636105] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636108] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636113] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636115] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636116] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636118] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636120] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636121] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636123] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636125] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636127] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636129] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636130] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636132] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636134] Stack: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636134] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636136] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636139] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636141] Call Trace: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636185] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636232] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636282] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636328] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636375] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636391] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636402] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636414] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636425] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636440] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636456] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636486] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636500] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636515] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636566] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636570] [] ? strlcpy+0x4a/0x60 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636586] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636601] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636615] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636630] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636674] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636714] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636768] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636823] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636876] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636928] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636932] [] ? thread_return+0x4e/0x7d0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636984] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636987] [] ? kthread+0x9e/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636990] [] ? child_rip+0xa/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636993] [] ? kthread+0x0/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636995] [] ? child_rip+0x0/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.636996] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637011] Call Trace: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637054] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637100] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637150] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637196] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637243] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637259] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637269] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637281] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637291] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637306] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637322] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637351] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637366] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637381] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637431] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637435] [] ? strlcpy+0x4a/0x60 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637451] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637466] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637481] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637496] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637539] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637580] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637634] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637688] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637741] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637793] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637797] [] ? thread_return+0x4e/0x7d0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637848] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637852] [] ? kthread+0x9e/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637854] [] ? child_rip+0xa/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637857] [] ? kthread+0x0/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.637859] [] ? child_rip+0x0/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640023] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640025] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640068] CPU 6 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640069] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640106] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640108] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640111] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640116] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640118] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640119] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640121] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640123] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640124] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640127] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640129] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640130] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640132] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640134] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640136] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640137] Stack: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640138] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640140] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640142] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640145] Call Trace: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640180] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640226] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640276] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640323] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640370] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640385] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640396] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640408] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640420] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640470] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640481] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640497] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640528] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640545] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640573] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640588] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640603] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640654] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640658] [] ? strlcpy+0x4a/0x60 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640674] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640689] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640704] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640718] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640762] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640803] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640857] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640911] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.640964] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641016] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641020] [] ? thread_return+0x4e/0x7d0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641072] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641075] [] ? kthread+0x9e/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641078] [] ? child_rip+0xa/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641081] [] ? kthread+0x0/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641083] [] ? child_rip+0x0/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641084] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641100] Call Trace: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641142] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641189] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641238] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641285] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641332] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641347] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641358] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641369] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641382] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641432] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641442] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641458] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641487] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641503] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641532] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641546] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641561] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641612] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641615] [] ? strlcpy+0x4a/0x60 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641632] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641646] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641661] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641676] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641720] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641760] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641814] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641868] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641922] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641974] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.641977] [] ? thread_return+0x4e/0x7d0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.642029] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.642033] [] ? kthread+0x9e/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.642035] [] ? child_rip+0xa/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.642038] [] ? kthread+0x0/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.642040] [] ? child_rip+0x0/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643023] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643024] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643064] CPU 7 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643065] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643100] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643102] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643105] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643109] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643111] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643112] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643114] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643115] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643117] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643119] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643121] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643122] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643125] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643127] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643129] Stack: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643129] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643131] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643134] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643136] Call Trace: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643170] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643217] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643267] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643313] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643360] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643376] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643386] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643398] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643409] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643424] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643440] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643469] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643484] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643499] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643549] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643553] [] ? strlcpy+0x4a/0x60 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643569] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643584] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643599] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643614] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643658] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643698] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643752] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643806] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643859] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643911] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643914] [] ? thread_return+0x4e/0x7d0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643966] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643970] [] ? kthread+0x9e/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643972] [] ? child_rip+0xa/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643975] [] ? kthread+0x0/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643977] [] ? child_rip+0x0/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643979] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.643994] Call Trace: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644037] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644083] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644133] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644179] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644226] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644241] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644252] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644264] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644274] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644289] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644305] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644334] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644349] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644364] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644414] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644418] [] ? strlcpy+0x4a/0x60 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644434] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644449] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644464] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644478] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644522] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644562] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644616] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644670] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644724] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644776] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644779] [] ? thread_return+0x4e/0x7d0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644831] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644834] [] ? kthread+0x9e/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644837] [] ? child_rip+0xa/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644840] [] ? kthread+0x0/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477629.644842] [] ? child_rip+0x0/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.149415] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.251330] CPU 4 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.253446] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.350872] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.352728] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.366048] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.374654] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.380522] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.388347] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.396208] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.404062] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.411956] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.419879] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.428728] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.435037] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.442914] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.450794] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.458628] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.468032] Stack: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.470424] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.478329] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.486786] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.495584] Call Trace: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.498493] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.505908] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.513980] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.521587] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.529031] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.536548] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.544162] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.551531] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.558559] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.565444] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.573307] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.580418] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.588655] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.594981] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.601943] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.609860] [] ? strlcpy+0x4a/0x60 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.615498] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.622309] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.629658] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.636816] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.644134] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.651742] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.659779] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.667114] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.674063] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.681975] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.689051] [] ? thread_return+0x4e/0x7d0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.695357] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.702294] [] ? kthread+0x9e/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.707840] [] ? child_rip+0xa/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.713491] [] ? kthread+0x0/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.718985] [] ? child_rip+0x0/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.724644] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.746129] Call Trace: Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.749021] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.756367] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.764387] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.772010] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.779464] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.787000] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.794630] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.801989] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.809018] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.815930] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.823722] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.830877] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.839070] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.845423] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.852392] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.860318] [] ? strlcpy+0x4a/0x60 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.865948] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.872759] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.880117] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.887241] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.894566] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.902069] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.910057] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.917422] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.924331] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.932155] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.939292] [] ? thread_return+0x4e/0x7d0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.945625] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.952500] [] ? kthread+0x9e/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.958106] [] ? child_rip+0xa/0x20 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.963802] [] ? kthread+0x0/0xc0 Aug 25 18:42:51 feral28.ccs.ornl.gov kernel: [1477632.969277] [] ? child_rip+0x0/0x20 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.834065] INFO: task crond:17243 blocked for more than 120 seconds. Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.841094] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.850339] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.858927] crond D 0000000000000003 0 17243 11135 0x00000000 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.866518] ffff880bbe307d38 0000000000000082 0000000000000000 0000000000000000 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.874740] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.882895] 0000000000000000 0000000000000000 ffff880bd810c5f8 ffff880bbe307fd8 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.891097] Call Trace: Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.893990] [] schedule_timeout+0x215/0x2e0 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.900386] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.906900] [] wait_for_common+0x123/0x180 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.913194] [] ? default_wake_function+0x0/0x20 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.919952] [] wait_for_completion+0x1d/0x20 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.926419] [] synchronize_sched+0x58/0x60 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.932700] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.938962] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.946559] [] join_session_keyring+0x133/0x160 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.953321] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.960468] [] sys_keyctl+0x180/0x1a0 Aug 25 18:43:37 feral28.ccs.ornl.gov kernel: [1477678.966270] [] system_call_fastpath+0x16/0x1b Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.639642] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642642] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642644] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642687] CPU 5 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642687] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642724] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642726] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642729] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642733] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642735] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642737] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642738] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642740] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642742] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642744] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642746] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642747] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642749] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642751] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642753] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642754] Stack: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642755] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642757] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642759] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642762] Call Trace: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642806] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642852] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642902] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642949] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.642995] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643011] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643022] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643034] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643045] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643060] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643076] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643106] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643121] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643136] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643186] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643190] [] ? strlcpy+0x4a/0x60 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643206] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643221] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643236] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643251] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643294] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643335] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643389] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643443] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643497] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643549] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643552] [] ? thread_return+0x4e/0x7d0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643604] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643607] [] ? kthread+0x9e/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643610] [] ? child_rip+0xa/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643613] [] ? kthread+0x0/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643615] [] ? child_rip+0x0/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643616] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643632] Call Trace: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643674] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643721] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643770] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643817] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643864] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643879] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643890] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643902] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643912] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643927] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643943] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643972] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.643987] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644002] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644052] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644056] [] ? strlcpy+0x4a/0x60 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644072] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644087] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644101] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644116] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644160] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644200] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644254] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644309] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644362] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644414] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644418] [] ? thread_return+0x4e/0x7d0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644469] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644473] [] ? kthread+0x9e/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644476] [] ? child_rip+0xa/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644479] [] ? kthread+0x0/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.644481] [] ? child_rip+0x0/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646642] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646644] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646687] CPU 6 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646688] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646724] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646725] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646728] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646733] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646735] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646736] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646738] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646739] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646741] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646743] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646745] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646746] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646748] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646750] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646752] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646753] Stack: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646754] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646756] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646758] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646761] Call Trace: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646796] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646843] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646893] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646939] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.646986] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647002] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647013] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647024] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647037] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647087] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647097] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647113] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647144] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647161] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647189] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647204] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647219] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647269] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647273] [] ? strlcpy+0x4a/0x60 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647289] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647304] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647319] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647334] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647378] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647418] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647472] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647526] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647580] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647632] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647635] [] ? thread_return+0x4e/0x7d0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647687] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647690] [] ? kthread+0x9e/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647693] [] ? child_rip+0xa/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647696] [] ? kthread+0x0/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647698] [] ? child_rip+0x0/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647699] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647715] Call Trace: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647757] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647804] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647853] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647900] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647947] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647962] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647973] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647985] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.647997] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648047] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648057] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648073] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648102] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648118] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648147] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648161] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648177] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648227] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648230] [] ? strlcpy+0x4a/0x60 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648247] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648261] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648276] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648291] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648335] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648375] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648429] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648483] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648536] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648588] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648592] [] ? thread_return+0x4e/0x7d0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648643] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648647] [] ? kthread+0x9e/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648650] [] ? child_rip+0xa/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648652] [] ? kthread+0x0/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.648655] [] ? child_rip+0x0/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649641] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649643] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649682] CPU 7 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649683] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649719] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649720] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649723] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649727] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649729] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649731] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649732] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649734] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649735] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649737] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649739] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649741] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649743] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649744] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649746] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649748] Stack: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649749] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649751] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649753] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649756] Call Trace: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649790] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649836] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649886] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649932] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649979] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.649995] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650005] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650017] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650028] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650043] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650059] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650088] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650103] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650118] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650168] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650172] [] ? strlcpy+0x4a/0x60 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650188] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650203] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650218] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650232] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650276] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650317] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650371] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650425] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650478] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650530] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650534] [] ? thread_return+0x4e/0x7d0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650585] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650589] [] ? kthread+0x9e/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650592] [] ? child_rip+0xa/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650594] [] ? kthread+0x0/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650597] [] ? child_rip+0x0/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650598] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650613] Call Trace: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650656] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650702] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650752] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650798] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650845] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650861] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650871] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650883] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650894] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650909] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650925] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650954] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650968] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.650983] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651033] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651037] [] ? strlcpy+0x4a/0x60 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651054] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651068] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651083] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651098] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651142] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651182] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651236] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651290] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651343] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651395] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651399] [] ? thread_return+0x4e/0x7d0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651450] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651454] [] ? kthread+0x9e/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651457] [] ? child_rip+0xa/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651460] [] ? kthread+0x0/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477713.651462] [] ? child_rip+0x0/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.170265] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.267351] CPU 4 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.269474] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.367057] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.368912] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.382224] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.390935] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.396836] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.404750] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.412576] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.420524] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.428453] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.436376] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.445319] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.451654] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.459604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.467525] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.475438] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.484910] Stack: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.487319] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.495207] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.503701] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.512523] Call Trace: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.515409] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.522804] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.530799] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.538457] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.545894] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.553403] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.560989] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.568393] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.575542] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.582402] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.590229] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.597435] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.605776] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.612147] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.619169] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.627113] [] ? strlcpy+0x4a/0x60 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.632758] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.639552] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.646901] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.654086] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.661387] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.668958] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.677058] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.684451] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.691401] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.699235] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.706355] [] ? thread_return+0x4e/0x7d0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.712652] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.719511] [] ? kthread+0x9e/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.725152] [] ? child_rip+0xa/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.730871] [] ? kthread+0x0/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.736375] [] ? child_rip+0x0/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.742069] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.763485] Call Trace: Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.766421] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.773729] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.781784] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.789497] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.796991] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.804511] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.812220] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.819589] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.826693] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.833614] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.841422] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.848622] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.856937] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.866055] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.873032] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.880968] [] ? strlcpy+0x4a/0x60 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.886606] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.893434] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.900783] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.907942] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.915303] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.922883] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.930975] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.938402] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.945318] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.953246] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.960306] [] ? thread_return+0x4e/0x7d0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.966620] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.973497] [] ? kthread+0x9e/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.979086] [] ? child_rip+0xa/0x20 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.984789] [] ? kthread+0x0/0xc0 Aug 25 18:44:15 feral28.ccs.ornl.gov kernel: [1477716.990317] [] ? child_rip+0x0/0x20 Aug 25 18:44:57 feral28.ccs.ornl.gov kernel: [1477759.658391] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 18:44:57 feral28.ccs.ornl.gov kernel: [1477759.669422] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 3 previous similar messages Aug 25 18:44:57 feral28.ccs.ornl.gov kernel: [1477759.680464] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.101@o2ib1 (201): c: 63, oc: 0, rc: 63 Aug 25 18:44:57 feral28.ccs.ornl.gov kernel: [1477759.693965] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 3 previous similar messages Aug 25 18:45:29 feral28.ccs.ornl.gov kernel: [1477790.851728] Lustre: sultan-MDT000b: Client 121a88f7-96bf-42c6-fa63-63b545c0ecc1 (at 7@gni1) reconnecting Aug 25 18:45:29 feral28.ccs.ornl.gov kernel: [1477790.862080] Lustre: sultan-MDT000b: Connection restored to c532693a-b694-08e2-01f7-59677b3e0b03 (at 7@gni1) Aug 25 18:45:31 feral28.ccs.ornl.gov kernel: [1477793.283010] Lustre: sultan-MDT000b: Client f680e5fb-3752-3b31-4864-2e3f707d1adc (at 12@gni1) reconnecting Aug 25 18:45:31 feral28.ccs.ornl.gov kernel: [1477793.293359] Lustre: sultan-MDT000b: Connection restored to 560c2e5a-a98f-72ed-fa5f-899936c4b413 (at 12@gni1) Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.646261] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649261] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649262] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649305] CPU 5 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649306] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649342] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649343] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649346] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649351] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649352] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649354] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649355] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649357] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649358] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649360] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649362] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649364] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649365] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649367] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649369] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649370] Stack: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649371] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649373] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649375] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649378] Call Trace: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649414] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649460] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649510] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649557] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649603] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649619] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649630] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649642] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649653] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649668] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649685] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649714] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649729] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649744] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649794] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649798] [] ? strlcpy+0x4a/0x60 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649814] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649829] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649844] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649858] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649903] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649943] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.649997] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650051] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650105] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650157] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650160] [] ? thread_return+0x4e/0x7d0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650212] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650216] [] ? kthread+0x9e/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650218] [] ? child_rip+0xa/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650221] [] ? kthread+0x0/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650223] [] ? child_rip+0x0/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650224] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650240] Call Trace: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650282] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650329] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650378] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650425] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650472] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650487] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650497] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650509] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650520] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650535] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650551] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650580] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650595] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650609] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650660] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650664] [] ? strlcpy+0x4a/0x60 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650680] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650695] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650710] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650724] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650768] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650808] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650863] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650917] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.650970] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.651022] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.651026] [] ? thread_return+0x4e/0x7d0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.651078] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.651081] [] ? kthread+0x9e/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.651084] [] ? child_rip+0xa/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.651087] [] ? kthread+0x0/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.651089] [] ? child_rip+0x0/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653261] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653262] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653305] CPU 6 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653306] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653344] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653346] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653348] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653353] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653355] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653356] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653358] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653359] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653361] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653363] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653365] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653367] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653369] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653371] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653373] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653374] Stack: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653375] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653377] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653379] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653382] Call Trace: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653417] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653463] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653513] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653560] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653607] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653622] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653633] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653645] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653657] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653707] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653718] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653734] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653765] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653781] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653810] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653825] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653840] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653890] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653894] [] ? strlcpy+0x4a/0x60 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653910] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653925] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653940] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653955] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.653999] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654039] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654094] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654148] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654201] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654253] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654257] [] ? thread_return+0x4e/0x7d0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654308] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654312] [] ? kthread+0x9e/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654315] [] ? child_rip+0xa/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654318] [] ? kthread+0x0/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654320] [] ? child_rip+0x0/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654321] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654337] Call Trace: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654379] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654426] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654475] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654522] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654568] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654584] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654595] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654606] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654618] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654669] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654679] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654694] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654724] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654740] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654768] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654783] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654798] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654848] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654852] [] ? strlcpy+0x4a/0x60 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654868] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654883] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654898] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654913] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654956] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.654997] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655051] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655105] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655158] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655210] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655213] [] ? thread_return+0x4e/0x7d0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655266] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655269] [] ? kthread+0x9e/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655272] [] ? child_rip+0xa/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655275] [] ? kthread+0x0/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.655277] [] ? child_rip+0x0/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656260] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656261] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656304] CPU 7 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656305] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656341] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656343] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656345] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656349] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656351] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656353] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656355] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656356] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656358] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656360] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656362] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656364] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656366] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656367] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656369] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656370] Stack: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656371] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656373] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656375] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656378] Call Trace: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656412] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656458] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656508] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656555] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656601] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656617] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656628] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656640] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656651] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656666] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656682] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656711] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656726] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656741] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656791] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656795] [] ? strlcpy+0x4a/0x60 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656811] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656826] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656841] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656856] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656900] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656940] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.656994] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657048] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657102] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657153] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657157] [] ? thread_return+0x4e/0x7d0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657209] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657212] [] ? kthread+0x9e/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657215] [] ? child_rip+0xa/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657218] [] ? kthread+0x0/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657220] [] ? child_rip+0x0/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657221] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657237] Call Trace: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657279] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657326] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657375] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657422] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657469] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657484] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657495] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657506] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657517] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657532] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657548] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657577] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657591] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657606] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657656] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657660] [] ? strlcpy+0x4a/0x60 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657677] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657691] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657706] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657721] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657765] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657805] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657859] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657913] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.657967] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.658018] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.658022] [] ? thread_return+0x4e/0x7d0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.658074] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.658077] [] ? kthread+0x9e/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.658080] [] ? child_rip+0xa/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.658083] [] ? kthread+0x0/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477797.658085] [] ? child_rip+0x0/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982636] INFO: task crond:17243 blocked for more than 120 seconds. Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982641] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982643] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982647] crond D 0000000000000003 0 17243 11135 0x00000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982653] ffff880bbe307d38 0000000000000082 0000000000000000 0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982657] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982661] 0000000000000000 0000000000000000 ffff880bd810c5f8 ffff880bbe307fd8 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982665] Call Trace: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982677] [] schedule_timeout+0x215/0x2e0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982686] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982691] [] wait_for_common+0x123/0x180 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982702] [] ? default_wake_function+0x0/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982704] [] wait_for_completion+0x1d/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982709] [] synchronize_sched+0x58/0x60 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982711] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982716] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982719] [] join_session_keyring+0x133/0x160 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982722] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982725] [] sys_keyctl+0x180/0x1a0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477798.982730] [] system_call_fastpath+0x16/0x1b Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.300870] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.398020] CPU 4 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.400126] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.497361] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.499208] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.512607] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.521247] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.527132] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.535010] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.542922] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.553523] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.561384] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.569270] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.578077] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.584387] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.592204] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.600081] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.607942] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.617370] Stack: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.619781] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.627660] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.636144] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.644968] Call Trace: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.647875] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.655648] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.664026] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.671759] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.679714] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.687381] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.695106] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.702900] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.710099] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.717047] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.725161] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.732540] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.741089] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.747666] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.754854] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.763130] [] ? strlcpy+0x4a/0x60 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.768966] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.776060] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.783696] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.791245] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.798754] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.806542] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.814659] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.822094] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.829027] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.836906] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.843946] [] ? thread_return+0x4e/0x7d0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.850296] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.857173] [] ? kthread+0x9e/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.862736] [] ? child_rip+0xa/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.868412] [] ? kthread+0x0/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.873864] [] ? child_rip+0x0/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.879514] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.901120] Call Trace: Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.904030] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.911358] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.919352] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.927002] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.934421] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.941922] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.949500] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.956816] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.963852] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.970773] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.978558] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.985712] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477800.993974] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.000309] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.007304] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.015214] [] ? strlcpy+0x4a/0x60 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.020852] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.027689] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.035126] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.042293] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.049655] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.057226] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.065342] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.072728] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.079661] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.087537] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.094682] [] ? thread_return+0x4e/0x7d0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.100979] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.107856] [] ? kthread+0x9e/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.113446] [] ? child_rip+0xa/0x20 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.119149] [] ? kthread+0x0/0xc0 Aug 25 18:45:39 feral28.ccs.ornl.gov kernel: [1477801.124651] [] ? child_rip+0x0/0x20 Aug 25 18:45:49 feral28.ccs.ornl.gov kernel: [1477811.442703] Lustre: sultan-MDT000b: Client 1f0727a3-d49c-044b-6def-11f5d586ab1d (at 26@gni1) reconnecting Aug 25 18:45:49 feral28.ccs.ornl.gov kernel: [1477811.453117] Lustre: sultan-MDT000b: Connection restored to c60f26de-f3d6-6cfc-b87c-120a7b1f0eed (at 26@gni1) Aug 25 18:46:06 feral28.ccs.ornl.gov kernel: [1477828.026336] Lustre: sultan-MDT000b: Client ade025d5-35bd-119e-3200-b4438b686c43 (at 20@gni1) reconnecting Aug 25 18:46:06 feral28.ccs.ornl.gov kernel: [1477828.036846] Lustre: sultan-MDT000b: Connection restored to 84595275-af73-a989-8d49-91a86f054b0f (at 20@gni1) Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.652879] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655879] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655880] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655924] CPU 5 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655925] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655961] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655963] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655966] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655970] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655972] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655974] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655975] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655977] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655978] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655980] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655982] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655984] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655986] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655987] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655989] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655991] Stack: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655992] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655994] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655996] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.655999] Call Trace: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656043] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656089] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656139] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656185] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656232] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656248] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656259] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656271] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656282] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656297] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656313] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656343] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656357] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656372] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656423] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656427] [] ? strlcpy+0x4a/0x60 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656443] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656458] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656473] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656488] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656531] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656572] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656626] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656680] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656734] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656785] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656789] [] ? thread_return+0x4e/0x7d0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656841] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656844] [] ? kthread+0x9e/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656847] [] ? child_rip+0xa/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656850] [] ? kthread+0x0/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656852] [] ? child_rip+0x0/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656853] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656868] Call Trace: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656911] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.656957] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657007] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657054] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657101] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657116] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657127] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657139] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657149] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657164] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657180] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657209] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657224] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657239] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657289] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657293] [] ? strlcpy+0x4a/0x60 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657309] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657324] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657339] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657353] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657397] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657438] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657492] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657546] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657599] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657651] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657655] [] ? thread_return+0x4e/0x7d0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657706] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657710] [] ? kthread+0x9e/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657713] [] ? child_rip+0xa/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657715] [] ? kthread+0x0/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.657718] [] ? child_rip+0x0/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659879] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659881] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659924] CPU 6 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659925] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659961] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659963] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659966] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659971] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659973] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659974] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659976] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659978] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659980] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659982] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659984] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659986] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659989] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659991] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659993] Stack: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659993] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659995] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.659998] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660000] Call Trace: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660035] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660081] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660131] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660178] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660224] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660240] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660251] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660263] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660275] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660325] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660336] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660351] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660381] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660397] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660426] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660441] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660456] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660506] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660510] [] ? strlcpy+0x4a/0x60 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660526] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660541] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660556] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660571] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660615] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660655] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660709] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660762] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660816] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660868] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660871] [] ? thread_return+0x4e/0x7d0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660923] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660927] [] ? kthread+0x9e/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660929] [] ? child_rip+0xa/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660932] [] ? kthread+0x0/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660934] [] ? child_rip+0x0/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660935] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660951] Call Trace: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.660994] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661040] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661090] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661136] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661183] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661198] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661209] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661220] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661233] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661283] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661293] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661308] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661337] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661354] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661382] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661397] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661412] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661462] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661466] [] ? strlcpy+0x4a/0x60 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661482] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661497] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661512] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661526] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661570] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661611] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661664] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661718] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661772] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661824] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661827] [] ? thread_return+0x4e/0x7d0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661879] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661882] [] ? kthread+0x9e/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661885] [] ? child_rip+0xa/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661888] [] ? kthread+0x0/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.661890] [] ? child_rip+0x0/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662878] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662880] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662920] CPU 7 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662921] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662957] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662959] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662962] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662966] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662967] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662969] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662971] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662972] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662974] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662976] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662977] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662979] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662982] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662984] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662985] Stack: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662986] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662988] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662991] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.662993] Call Trace: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663027] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663074] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663123] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663170] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663217] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663232] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663243] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663255] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663265] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663281] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663297] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663326] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663341] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663356] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663406] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663410] [] ? strlcpy+0x4a/0x60 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663426] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663441] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663456] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663470] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663514] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663555] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663609] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663663] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663716] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663768] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663772] [] ? thread_return+0x4e/0x7d0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663823] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663827] [] ? kthread+0x9e/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663830] [] ? child_rip+0xa/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663832] [] ? kthread+0x0/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663835] [] ? child_rip+0x0/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663836] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663851] Call Trace: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663894] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663940] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.663990] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664036] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664083] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664099] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664109] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664121] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664132] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664147] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664162] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664191] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664206] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664221] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664271] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664275] [] ? strlcpy+0x4a/0x60 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664291] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664306] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664321] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664335] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664379] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664420] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664473] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664527] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664581] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664632] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664636] [] ? thread_return+0x4e/0x7d0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664688] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664691] [] ? kthread+0x9e/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664694] [] ? child_rip+0xa/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664697] [] ? kthread+0x0/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477881.664699] [] ? child_rip+0x0/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.182735] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.279711] CPU 4 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.281817] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.379312] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.381176] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.394625] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.403268] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.409127] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.417047] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.424996] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.432891] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.440839] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.448744] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.457696] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.464006] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.471962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.479883] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.487831] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.497286] Stack: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.499678] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.507566] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.515964] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.524813] Call Trace: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.527759] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.535146] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.543251] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.550894] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.558391] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.565944] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.573616] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.580986] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.588083] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.594978] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.602805] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.610012] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.618309] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.624644] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.631701] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.639670] [] ? strlcpy+0x4a/0x60 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.645334] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.652223] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.659615] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.666774] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.674014] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.681610] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.689669] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.697122] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.704010] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.711897] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.719042] [] ? thread_return+0x4e/0x7d0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.725340] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.732251] [] ? kthread+0x9e/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.737806] [] ? child_rip+0xa/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.743500] [] ? kthread+0x0/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.749046] [] ? child_rip+0x0/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.754696] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.776251] Call Trace: Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.779108] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.786470] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.794456] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.802071] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.809508] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.817060] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.824638] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.831981] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.839034] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.845885] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.853661] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.860773] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.869018] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.875336] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.882304] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.890128] [] ? strlcpy+0x4a/0x60 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.895740] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.902525] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.909918] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.917060] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.924323] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.931914] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.939962] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.947373] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.954322] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.962235] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.969327] [] ? thread_return+0x4e/0x7d0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.975615] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.982508] [] ? kthread+0x9e/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.988115] [] ? child_rip+0xa/0x20 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.993775] [] ? kthread+0x0/0xc0 Aug 25 18:47:03 feral28.ccs.ornl.gov kernel: [1477884.999286] [] ? child_rip+0x0/0x20 Aug 25 18:47:08 feral28.ccs.ornl.gov kernel: [1477890.668707] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 18:47:08 feral28.ccs.ornl.gov kernel: [1477890.679720] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 11 previous similar messages Aug 25 18:47:08 feral28.ccs.ornl.gov kernel: [1477890.690807] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.119@o2ib1 (203): c: 63, oc: 0, rc: 63 Aug 25 18:47:08 feral28.ccs.ornl.gov kernel: [1477890.704254] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 11 previous similar messages Aug 25 18:47:26 feral28.ccs.ornl.gov kernel: [1477908.304576] Lustre: sultan-MDT000b: Client 1561233c-aa4b-81d5-e0e4-de3b3609634b (at 2@gni1) reconnecting Aug 25 18:47:26 feral28.ccs.ornl.gov kernel: [1477908.314961] Lustre: sultan-MDT000b: Connection restored to 283d05d7-934b-fbcd-89aa-de4f53ec8e22 (at 2@gni1) Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477918.992087] INFO: task crond:17243 blocked for more than 120 seconds. Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477918.999074] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.008291] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.016880] crond D 0000000000000003 0 17243 11135 0x00000000 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.024376] ffff880bbe307d38 0000000000000082 0000000000000000 0000000000000000 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.032587] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.040725] 0000000000000000 0000000000000000 ffff880bd810c5f8 ffff880bbe307fd8 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.048892] Call Trace: Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.051737] [] schedule_timeout+0x215/0x2e0 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.058103] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.064610] [] wait_for_common+0x123/0x180 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.070913] [] ? default_wake_function+0x0/0x20 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.080314] [] wait_for_completion+0x1d/0x20 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.086798] [] synchronize_sched+0x58/0x60 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.093124] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.099435] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.106947] [] join_session_keyring+0x133/0x160 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.113614] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.120826] [] sys_keyctl+0x180/0x1a0 Aug 25 18:47:37 feral28.ccs.ornl.gov kernel: [1477919.126665] [] system_call_fastpath+0x16/0x1b Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.659498] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662498] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662499] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662542] CPU 5 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662543] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662579] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662581] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662584] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662588] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662590] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662592] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662593] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662595] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662596] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662599] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662601] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662602] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662604] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662606] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662608] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662609] Stack: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662610] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662612] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662614] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662617] Call Trace: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662661] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662707] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662757] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662804] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662850] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662866] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662877] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662889] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662900] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662915] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662931] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662961] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662975] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.662990] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663041] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663045] [] ? strlcpy+0x4a/0x60 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663062] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663076] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663091] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663106] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663150] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663190] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663244] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663299] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663352] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663404] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663407] [] ? thread_return+0x4e/0x7d0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663459] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663463] [] ? kthread+0x9e/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663465] [] ? child_rip+0xa/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663468] [] ? kthread+0x0/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663470] [] ? child_rip+0x0/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663472] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663487] Call Trace: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663530] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663576] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663626] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663672] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663719] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663735] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663745] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663757] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663768] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663783] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663799] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663828] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663842] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663857] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663907] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663911] [] ? strlcpy+0x4a/0x60 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663928] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663942] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663957] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.663972] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664016] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664056] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664110] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664164] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664218] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664270] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664273] [] ? thread_return+0x4e/0x7d0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664325] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664328] [] ? kthread+0x9e/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664331] [] ? child_rip+0xa/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664334] [] ? kthread+0x0/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.664336] [] ? child_rip+0x0/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666498] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666500] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666543] CPU 6 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666544] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666581] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666582] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666585] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666590] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666592] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666593] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666595] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666596] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666598] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666600] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666602] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666604] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666608] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666610] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666611] Stack: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666612] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666614] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666616] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666619] Call Trace: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666654] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666700] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666750] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666796] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666843] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666859] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666870] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666882] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666894] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666944] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666955] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.666970] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667001] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667018] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667046] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667061] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667076] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667126] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667130] [] ? strlcpy+0x4a/0x60 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667147] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667161] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667176] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667191] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667235] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667275] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667329] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667383] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667437] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667488] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667492] [] ? thread_return+0x4e/0x7d0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667543] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667547] [] ? kthread+0x9e/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667550] [] ? child_rip+0xa/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667552] [] ? kthread+0x0/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667554] [] ? child_rip+0x0/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667556] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667571] Call Trace: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667614] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667660] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667710] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667756] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667803] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667818] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667829] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667841] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667853] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667903] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667914] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667929] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667958] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.667975] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668004] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668019] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668034] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668085] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668089] [] ? strlcpy+0x4a/0x60 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668105] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668120] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668135] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668149] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668193] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668233] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668287] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668341] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668394] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668446] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668450] [] ? thread_return+0x4e/0x7d0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668502] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668505] [] ? kthread+0x9e/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668508] [] ? child_rip+0xa/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668511] [] ? kthread+0x0/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.668513] [] ? child_rip+0x0/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669497] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669498] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669538] CPU 7 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669539] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669575] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669577] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669580] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669584] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669585] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669587] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669589] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669590] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669592] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669594] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669596] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669597] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669601] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669603] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669604] Stack: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669605] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669607] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669610] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669612] Call Trace: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669646] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669692] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669742] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669788] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669835] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669851] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669861] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669873] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669884] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669899] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669915] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669944] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669959] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.669974] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670024] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670028] [] ? strlcpy+0x4a/0x60 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670044] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670059] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670074] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670088] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670132] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670173] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670227] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670281] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670334] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670386] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670390] [] ? thread_return+0x4e/0x7d0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670441] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670445] [] ? kthread+0x9e/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670448] [] ? child_rip+0xa/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670451] [] ? kthread+0x0/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670453] [] ? child_rip+0x0/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670454] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670469] Call Trace: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670512] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670558] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670608] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670654] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670701] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670717] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670727] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670739] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670750] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670765] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670781] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670809] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670824] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670839] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670889] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670893] [] ? strlcpy+0x4a/0x60 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670909] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670924] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670939] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670954] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.670997] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671037] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671092] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671146] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671199] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671251] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671254] [] ? thread_return+0x4e/0x7d0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671306] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671309] [] ? kthread+0x9e/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671312] [] ? child_rip+0xa/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671315] [] ? kthread+0x0/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477965.671317] [] ? child_rip+0x0/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477966.498920] Lustre: sultan-MDT000b: Client 1f0727a3-d49c-044b-6def-11f5d586ab1d (at 26@gni1) reconnecting Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477966.498959] Lustre: sultan-MDT000b: Connection restored to c60f26de-f3d6-6cfc-b87c-120a7b1f0eed (at 26@gni1) Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.212832] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.310179] CPU 4 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.312277] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.409790] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.411628] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.425069] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.433745] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.439604] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.447569] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.455431] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.463377] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.471282] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.479203] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.488112] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.494449] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.502379] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.510274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.518179] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.527652] Stack: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.530061] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.537965] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.546476] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.555307] Call Trace: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.558210] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.565606] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.573670] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.581380] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.588842] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.596343] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.603956] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.611307] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.618378] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.625230] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.633074] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.640306] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.648578] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.654905] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.661935] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.669862] [] ? strlcpy+0x4a/0x60 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.675516] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.682301] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.689668] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.696879] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.704119] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.711604] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.719702] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.729757] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.736734] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.744610] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.751730] [] ? thread_return+0x4e/0x7d0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.757923] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.764791] [] ? kthread+0x9e/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.770459] [] ? child_rip+0xa/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.776153] [] ? kthread+0x0/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.781682] [] ? child_rip+0x0/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.787444] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.808936] Call Trace: Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.811832] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.819193] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.827274] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.834949] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.842384] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.849886] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.857466] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.864807] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.871895] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.878756] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.886557] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.893788] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.902131] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.908466] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.915548] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.923691] [] ? strlcpy+0x4a/0x60 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.929337] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.936129] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.943481] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.950665] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.957938] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.965467] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.973472] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.980770] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.987658] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477968.995467] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477969.002578] [] ? thread_return+0x4e/0x7d0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477969.008910] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477969.015796] [] ? kthread+0x9e/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477969.021348] [] ? child_rip+0xa/0x20 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477969.026992] [] ? kthread+0x0/0xc0 Aug 25 18:48:27 feral28.ccs.ornl.gov kernel: [1477969.032469] [] ? child_rip+0x0/0x20 Aug 25 18:49:08 feral28.ccs.ornl.gov kernel: [1478010.518643] Lustre: sultan-MDT000b: haven't heard from client 8d48f2bb-3a94-5dec-2c5c-1f0ea8b2f1ab (at 25@gni1) in 342 seconds. I think it's dead, and I am evicting it. exp ffff880bc26b9400, cur 1472165348 expire 1472165126 last 1472165006 Aug 25 18:49:08 feral28.ccs.ornl.gov kernel: [1478010.541334] Lustre: Skipped 14 previous similar messages Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.142566] INFO: task crond:17243 blocked for more than 120 seconds. Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.149566] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.158765] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.167354] crond D 0000000000000003 0 17243 11135 0x00000000 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.174885] ffff880bbe307d38 0000000000000082 0000000000000000 0000000000000000 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.183089] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.191322] 0000000000000000 0000000000000000 ffff880bd810c5f8 ffff880bbe307fd8 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.199507] Call Trace: Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.202341] [] schedule_timeout+0x215/0x2e0 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.208717] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.215180] [] wait_for_common+0x123/0x180 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.221481] [] ? default_wake_function+0x0/0x20 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.228241] [] wait_for_completion+0x1d/0x20 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.234733] [] synchronize_sched+0x58/0x60 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.240998] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.247240] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.254822] [] join_session_keyring+0x133/0x160 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.261557] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.268715] [] sys_keyctl+0x180/0x1a0 Aug 25 18:49:37 feral28.ccs.ornl.gov kernel: [1478039.274507] [] system_call_fastpath+0x16/0x1b Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.666116] BUG: soft lockup - CPU#4 stuck for 68s! [mdt01_002:21487] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669116] BUG: soft lockup - CPU#5 stuck for 68s! [mdt01_007:17158] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669118] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669160] CPU 5 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669160] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669196] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669198] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669201] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669205] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669207] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669208] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669210] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669211] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669213] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669215] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669217] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669218] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669222] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669224] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669225] Stack: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669226] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669228] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669230] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669233] Call Trace: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669277] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669323] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669373] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669420] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669467] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669483] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669493] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669505] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669516] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669531] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669547] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669577] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669591] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669606] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669656] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669661] [] ? strlcpy+0x4a/0x60 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669677] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669691] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669706] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669721] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669765] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669806] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669860] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669914] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.669968] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670020] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670023] [] ? thread_return+0x4e/0x7d0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670075] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670078] [] ? kthread+0x9e/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670081] [] ? child_rip+0xa/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670084] [] ? kthread+0x0/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670086] [] ? child_rip+0x0/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670087] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670103] Call Trace: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670145] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670192] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670241] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670288] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670334] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670350] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670360] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670372] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670383] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670398] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670414] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670442] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670457] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670472] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670522] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670526] [] ? strlcpy+0x4a/0x60 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670542] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670557] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670572] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670586] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670630] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670670] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670724] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670778] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670832] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670884] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670887] [] ? thread_return+0x4e/0x7d0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670939] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670942] [] ? kthread+0x9e/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670945] [] ? child_rip+0xa/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670948] [] ? kthread+0x0/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.670950] [] ? child_rip+0x0/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673116] BUG: soft lockup - CPU#6 stuck for 68s! [mdt01_000:21485] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673118] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673160] CPU 6 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673161] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673198] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673200] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673203] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673208] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673210] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673211] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673213] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673215] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673216] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673219] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673221] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673222] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673224] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673226] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673228] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673229] Stack: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673230] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673232] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673234] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673237] Call Trace: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673271] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673318] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673368] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673414] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673461] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673477] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673488] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673500] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673512] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673562] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673573] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673588] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673619] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673636] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673664] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673679] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673694] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673744] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673749] [] ? strlcpy+0x4a/0x60 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673765] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673779] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673794] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673809] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673853] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673893] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.673947] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674001] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674055] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674107] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674110] [] ? thread_return+0x4e/0x7d0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674162] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674165] [] ? kthread+0x9e/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674168] [] ? child_rip+0xa/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674171] [] ? kthread+0x0/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674174] [] ? child_rip+0x0/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674175] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674190] Call Trace: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674233] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674279] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674329] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674376] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674422] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674438] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674449] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674460] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674472] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674522] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674533] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674548] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674577] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674594] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674622] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674637] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674652] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674702] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674706] [] ? strlcpy+0x4a/0x60 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674722] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674737] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674752] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674767] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674811] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674851] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674905] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.674959] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.675013] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.675064] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.675068] [] ? thread_return+0x4e/0x7d0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.675120] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.675123] [] ? kthread+0x9e/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.675126] [] ? child_rip+0xa/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.675128] [] ? kthread+0x0/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.675131] [] ? child_rip+0x0/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676115] BUG: soft lockup - CPU#7 stuck for 68s! [mdt01_001:21486] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676117] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676156] CPU 7 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676157] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676194] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676196] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676198] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676203] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676204] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676206] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676207] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676209] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676210] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676212] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676214] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676216] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676218] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676220] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676222] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676223] Stack: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676224] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676226] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676228] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676231] Call Trace: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676264] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676311] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676361] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676407] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676454] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676469] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676480] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676492] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676503] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676518] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676534] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676563] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676577] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676592] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676642] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676646] [] ? strlcpy+0x4a/0x60 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676663] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676677] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676692] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676707] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676751] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676792] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676846] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676900] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.676953] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677005] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677008] [] ? thread_return+0x4e/0x7d0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677060] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677064] [] ? kthread+0x9e/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677067] [] ? child_rip+0xa/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677069] [] ? kthread+0x0/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677071] [] ? child_rip+0x0/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677073] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677088] Call Trace: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677130] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677177] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677227] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677273] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677320] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677335] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677346] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677358] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677368] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677383] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677400] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677428] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677443] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677458] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677508] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677512] [] ? strlcpy+0x4a/0x60 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677528] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677543] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677558] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677573] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677616] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677657] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677711] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677765] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677818] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677870] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677874] [] ? thread_return+0x4e/0x7d0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677925] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677929] [] ? kthread+0x9e/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677932] [] ? child_rip+0xa/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677934] [] ? kthread+0x0/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478049.677937] [] ? child_rip+0x0/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.177744] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.274571] CPU 4 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.276660] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.376599] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.378438] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.391809] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.400362] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.406256] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.414109] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.421988] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.429882] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.437796] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.445623] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.454464] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.460781] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.468634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.476424] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.484320] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.493733] Stack: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.496142] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.503960] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.512445] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.521240] Call Trace: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.524126] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.531487] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.539473] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.547123] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.554569] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.562035] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.569665] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.576973] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.584018] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.590930] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.598723] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.605842] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.614044] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.620371] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.627330] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.635199] [] ? strlcpy+0x4a/0x60 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.640809] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.647655] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.655056] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.662224] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.669568] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.677181] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.685292] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.692684] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.699669] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.707538] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.714630] [] ? thread_return+0x4e/0x7d0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.720910] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.727847] [] ? kthread+0x9e/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.733438] [] ? child_rip+0xa/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.739148] [] ? kthread+0x0/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.744617] [] ? child_rip+0x0/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.750310] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.771839] Call Trace: Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.774722] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.782049] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.790045] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.797633] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.805069] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.812606] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.820219] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.827474] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.834553] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.841380] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.849206] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.856368] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.864589] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.870899] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.877912] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.885768] [] ? strlcpy+0x4a/0x60 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.891432] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.898287] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.905688] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.912872] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.920156] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.927700] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.935732] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.943106] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.950058] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.957882] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.964974] [] ? thread_return+0x4e/0x7d0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.971289] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.978165] [] ? kthread+0x9e/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.983731] [] ? child_rip+0xa/0x20 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.989399] [] ? kthread+0x0/0xc0 Aug 25 18:49:51 feral28.ccs.ornl.gov kernel: [1478052.994894] [] ? child_rip+0x0/0x20 Aug 25 18:50:28 feral28.ccs.ornl.gov kernel: [1478089.838544] Lustre: sultan-MDT000b: Client 165f3703-4728-d396-aa69-684cbccc927f (at 13@gni1) reconnecting Aug 25 18:50:28 feral28.ccs.ornl.gov kernel: [1478089.848949] Lustre: Skipped 1 previous similar message Aug 25 18:50:28 feral28.ccs.ornl.gov kernel: [1478089.854648] Lustre: sultan-MDT000b: Connection restored to bee067c8-71bc-8e18-b081-502ef15aafc5 (at 13@gni1) Aug 25 18:50:28 feral28.ccs.ornl.gov kernel: [1478089.865346] Lustre: Skipped 1 previous similar message Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.672735] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675735] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675736] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675779] CPU 5 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675780] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675815] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675817] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675820] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675824] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675826] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675828] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675829] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675831] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675832] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675834] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675836] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675838] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675841] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675843] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675844] Stack: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675845] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675847] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675849] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675852] Call Trace: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675896] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675942] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.675992] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676038] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676085] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676101] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676111] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676123] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676134] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676149] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676165] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676195] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676210] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676225] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676275] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676279] [] ? strlcpy+0x4a/0x60 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676295] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676310] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676325] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676340] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676384] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676424] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676478] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676532] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676586] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676638] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676642] [] ? thread_return+0x4e/0x7d0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676693] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676697] [] ? kthread+0x9e/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676699] [] ? child_rip+0xa/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676702] [] ? kthread+0x0/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676704] [] ? child_rip+0x0/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676706] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676721] Call Trace: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676764] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676810] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676860] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676906] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676953] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676968] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676979] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.676991] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677001] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677017] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677033] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677062] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677076] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677091] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677142] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677146] [] ? strlcpy+0x4a/0x60 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677162] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677176] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677191] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677206] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677250] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677290] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677344] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677398] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677452] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677503] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677507] [] ? thread_return+0x4e/0x7d0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677559] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677562] [] ? kthread+0x9e/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677565] [] ? child_rip+0xa/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677568] [] ? kthread+0x0/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.677570] [] ? child_rip+0x0/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679735] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679737] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679780] CPU 6 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679781] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679817] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679819] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679822] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679827] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679828] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679830] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679832] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679833] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679835] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679837] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679839] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679841] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679843] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679845] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679847] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679848] Stack: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679849] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679851] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679853] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679856] Call Trace: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679890] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679937] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.679986] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680033] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680080] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680095] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680106] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680118] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680130] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680181] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680191] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680207] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680238] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680255] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680283] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680298] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680313] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680364] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680367] [] ? strlcpy+0x4a/0x60 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680384] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680398] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680413] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680428] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680472] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680512] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680566] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680620] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680674] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680726] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680729] [] ? thread_return+0x4e/0x7d0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680781] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680784] [] ? kthread+0x9e/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680787] [] ? child_rip+0xa/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680790] [] ? kthread+0x0/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680792] [] ? child_rip+0x0/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680793] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680809] Call Trace: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680852] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680898] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680948] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.680994] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681041] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681056] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681067] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681079] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681091] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681141] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681152] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681167] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681198] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681214] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681243] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681257] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681272] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681323] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681327] [] ? strlcpy+0x4a/0x60 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681343] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681357] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681373] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681387] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681431] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681471] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681525] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681579] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681633] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681684] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681688] [] ? thread_return+0x4e/0x7d0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681740] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681743] [] ? kthread+0x9e/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681746] [] ? child_rip+0xa/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681749] [] ? kthread+0x0/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.681751] [] ? child_rip+0x0/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682734] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682735] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682775] CPU 7 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682776] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682813] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682815] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682818] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682822] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682823] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682825] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682827] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682829] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682830] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682832] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682834] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682836] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682838] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682840] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682841] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682843] Stack: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682844] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682846] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682848] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682850] Call Trace: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682884] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682930] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.682980] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683027] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683073] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683089] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683100] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683112] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683122] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683137] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683154] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683183] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683197] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683212] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683263] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683266] [] ? strlcpy+0x4a/0x60 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683283] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683298] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683313] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683327] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683371] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683412] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683466] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683520] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683573] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683625] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683628] [] ? thread_return+0x4e/0x7d0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683680] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683684] [] ? kthread+0x9e/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683687] [] ? child_rip+0xa/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683690] [] ? kthread+0x0/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683692] [] ? child_rip+0x0/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683693] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683708] Call Trace: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683751] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683797] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683847] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683893] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683940] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683955] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683966] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683978] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.683988] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684003] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684019] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684048] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684063] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684078] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684128] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684132] [] ? strlcpy+0x4a/0x60 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684148] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684163] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684177] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684192] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684236] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684276] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684330] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684384] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684438] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684490] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684493] [] ? thread_return+0x4e/0x7d0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684545] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684548] [] ? kthread+0x9e/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684551] [] ? child_rip+0xa/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684554] [] ? kthread+0x0/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478133.684556] [] ? child_rip+0x0/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.210542] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.307649] CPU 4 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.309781] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.407233] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.409070] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.422634] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.431421] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.437549] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.445548] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.453731] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.461782] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.469695] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.477661] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.486613] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.493044] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.501113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.509052] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.517433] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.526966] Stack: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.529453] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.537480] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.546101] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.555047] Call Trace: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.557933] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.565467] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.573556] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.581275] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.588738] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.596273] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.603982] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.611465] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.618577] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.625455] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.633213] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.640400] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.648621] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.654912] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.661882] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.669834] [] ? strlcpy+0x4a/0x60 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.675430] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.682273] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.689590] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.696723] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.704058] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.711586] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.719677] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.727070] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.734030] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.741975] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.749095] [] ? thread_return+0x4e/0x7d0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.755374] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.762260] [] ? kthread+0x9e/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.767919] [] ? child_rip+0xa/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.773665] [] ? kthread+0x0/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.779211] [] ? child_rip+0x0/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.784930] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.806293] Call Trace: Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.809204] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.816575] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.824604] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.832271] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.839698] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.847286] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.854900] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.862216] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.869374] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.876242] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.884061] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.891250] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.899486] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.905874] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.912905] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.920909] [] ? strlcpy+0x4a/0x60 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.926529] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.933392] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.940776] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.947961] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.955286] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.962851] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.970898] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.978377] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.985302] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478136.993197] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478137.000298] [] ? thread_return+0x4e/0x7d0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478137.006604] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478137.013472] [] ? kthread+0x9e/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478137.019104] [] ? child_rip+0xa/0x20 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478137.024797] [] ? kthread+0x0/0xc0 Aug 25 18:51:15 feral28.ccs.ornl.gov kernel: [1478137.030277] [] ? child_rip+0x0/0x20 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.290030] INFO: task crond:17243 blocked for more than 120 seconds. Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.297068] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.306318] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.314847] crond D 0000000000000003 0 17243 11135 0x00000000 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.322388] ffff880bbe307d38 0000000000000082 0000000000000000 0000000000000000 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.330564] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.338755] 0000000000000000 0000000000000000 ffff880bd810c5f8 ffff880bbe307fd8 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.349636] Call Trace: Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.352478] [] schedule_timeout+0x215/0x2e0 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.358819] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.365272] [] wait_for_common+0x123/0x180 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.371628] [] ? default_wake_function+0x0/0x20 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.378403] [] wait_for_completion+0x1d/0x20 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.384894] [] synchronize_sched+0x58/0x60 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.391206] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.397490] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.405089] [] join_session_keyring+0x133/0x160 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.411804] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.418990] [] sys_keyctl+0x180/0x1a0 Aug 25 18:51:37 feral28.ccs.ornl.gov kernel: [1478159.424858] [] system_call_fastpath+0x16/0x1b Aug 25 18:51:41 feral28.ccs.ornl.gov kernel: [1478163.311496] Lustre: sultan-MDT000b: haven't heard from client 5d9f30ae-edef-9a3d-c88a-aa425ee6b9cf (at 15@gni1) in 310 seconds. I think it's dead, and I am evicting it. exp ffff880bc25fd400, cur 1472165501 expire 1472165279 last 1472165191 Aug 25 18:51:41 feral28.ccs.ornl.gov kernel: [1478163.334234] Lustre: Skipped 1 previous similar message Aug 25 18:52:32 feral28.ccs.ornl.gov kernel: [1478214.693316] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 18:52:32 feral28.ccs.ornl.gov kernel: [1478214.704316] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 13 previous similar messages Aug 25 18:52:32 feral28.ccs.ornl.gov kernel: [1478214.715421] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.101@o2ib1 (212): c: 63, oc: 0, rc: 63 Aug 25 18:52:32 feral28.ccs.ornl.gov kernel: [1478214.728861] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 13 previous similar messages Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.679353] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682353] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682355] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682397] CPU 5 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682398] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682435] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682437] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682440] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682444] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682446] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682448] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682449] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682451] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682453] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682455] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682457] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682458] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682462] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682464] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682465] Stack: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682466] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682468] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682470] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682472] Call Trace: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682516] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682563] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682613] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682659] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682706] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682722] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682732] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682744] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682755] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682770] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682786] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682816] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682830] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682845] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682895] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682899] [] ? strlcpy+0x4a/0x60 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682916] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682930] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682945] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.682960] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683004] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683044] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683099] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683153] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683206] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683258] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683262] [] ? thread_return+0x4e/0x7d0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683314] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683317] [] ? kthread+0x9e/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683320] [] ? child_rip+0xa/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683323] [] ? kthread+0x0/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683325] [] ? child_rip+0x0/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683326] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683341] Call Trace: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683384] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683430] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683480] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683526] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683573] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683588] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683599] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683611] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683621] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683636] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683652] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683681] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683696] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683711] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683761] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683765] [] ? strlcpy+0x4a/0x60 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683781] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683796] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683810] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683825] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683869] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683910] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.683964] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.684018] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.684071] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.684123] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.684127] [] ? thread_return+0x4e/0x7d0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.684179] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.684182] [] ? kthread+0x9e/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.684185] [] ? child_rip+0xa/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.684188] [] ? kthread+0x0/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.684190] [] ? child_rip+0x0/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686353] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686355] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686398] CPU 6 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686399] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686436] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686438] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686441] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686445] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686447] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686449] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686450] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686452] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686453] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686456] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686457] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686459] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686461] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686462] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686465] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686466] Stack: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686467] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686469] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686471] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686474] Call Trace: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686509] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686556] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686605] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686652] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686699] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686715] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686725] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686737] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686750] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686800] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686811] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686826] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686857] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686874] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686903] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686917] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686932] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686983] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.686987] [] ? strlcpy+0x4a/0x60 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687003] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687018] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687033] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687047] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687091] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687132] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687186] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687240] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687293] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687345] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687349] [] ? thread_return+0x4e/0x7d0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687400] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687404] [] ? kthread+0x9e/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687407] [] ? child_rip+0xa/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687409] [] ? kthread+0x0/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687412] [] ? child_rip+0x0/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687413] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687428] Call Trace: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687471] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687517] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687567] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687613] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687660] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687675] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687686] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687698] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687710] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687760] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687771] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687786] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687817] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687833] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687862] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687877] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687892] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687942] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687946] [] ? strlcpy+0x4a/0x60 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687962] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687977] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.687992] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688007] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688050] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688098] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688153] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688206] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688260] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688312] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688315] [] ? thread_return+0x4e/0x7d0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688367] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688371] [] ? kthread+0x9e/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688374] [] ? child_rip+0xa/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688377] [] ? kthread+0x0/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.688379] [] ? child_rip+0x0/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689352] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689354] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689394] CPU 7 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689395] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689432] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689433] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689436] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689440] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689442] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689443] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689445] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689446] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689448] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689450] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689452] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689453] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689455] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689457] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689459] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689460] Stack: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689461] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689463] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689465] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689467] Call Trace: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689501] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689548] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689598] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689645] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689691] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689707] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689717] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689730] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689740] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689755] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689772] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689800] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689815] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689830] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689880] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689884] [] ? strlcpy+0x4a/0x60 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689900] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689915] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689930] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689945] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.689989] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690036] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690091] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690144] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690198] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690250] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690253] [] ? thread_return+0x4e/0x7d0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690305] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690309] [] ? kthread+0x9e/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690312] [] ? child_rip+0xa/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690314] [] ? kthread+0x0/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690317] [] ? child_rip+0x0/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690318] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690333] Call Trace: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690376] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690422] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690472] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690518] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690565] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690580] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690591] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690603] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690614] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690629] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690645] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690673] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690688] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690703] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690753] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690757] [] ? strlcpy+0x4a/0x60 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690773] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690788] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690803] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690817] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690861] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690902] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.690956] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.691010] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.691063] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.691115] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.691118] [] ? thread_return+0x4e/0x7d0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.691170] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.691173] [] ? kthread+0x9e/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.691176] [] ? child_rip+0xa/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.691179] [] ? kthread+0x0/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478217.691181] [] ? child_rip+0x0/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.191724] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.288715] CPU 4 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.290839] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.388160] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.390008] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.403379] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.412013] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.417861] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.425688] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.433515] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.441331] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.449218] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.457159] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.466040] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.472360] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.480194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.488047] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.495873] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.505337] Stack: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.507730] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.515661] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.524128] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.532924] Call Trace: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.535836] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.543205] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.551157] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.558763] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.566172] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.573664] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.581313] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.588664] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.595681] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.602534] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.610352] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.617516] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.628456] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.634722] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.641710] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.649559] [] ? strlcpy+0x4a/0x60 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.655144] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.661929] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.669279] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.676524] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.683849] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.691431] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.699496] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.706871] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.713831] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.721708] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.728740] [] ? thread_return+0x4e/0x7d0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.735029] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.741932] [] ? kthread+0x9e/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.747530] [] ? child_rip+0xa/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.753180] [] ? kthread+0x0/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.758690] [] ? child_rip+0x0/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.764376] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.785948] Call Trace: Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.788816] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.796143] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.804092] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.811743] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.819231] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.826741] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.834337] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.841584] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.848611] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.855480] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.863238] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.870367] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.878604] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.884859] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.891802] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.899706] [] ? strlcpy+0x4a/0x60 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.905316] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.912127] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.919486] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.926678] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.933954] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.941559] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.949574] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.957010] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.963942] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.971852] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.978912] [] ? thread_return+0x4e/0x7d0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.985251] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.992129] [] ? kthread+0x9e/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478220.997702] [] ? child_rip+0xa/0x20 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478221.003430] [] ? kthread+0x0/0xc0 Aug 25 18:52:39 feral28.ccs.ornl.gov kernel: [1478221.008959] [] ? child_rip+0x0/0x20 Aug 25 18:53:04 feral28.ccs.ornl.gov kernel: [1478246.604879] Lustre: sultan-MDT000b: Client 1f0727a3-d49c-044b-6def-11f5d586ab1d (at 26@gni1) reconnecting Aug 25 18:53:04 feral28.ccs.ornl.gov kernel: [1478246.615371] Lustre: Skipped 1 previous similar message Aug 25 18:53:04 feral28.ccs.ornl.gov kernel: [1478246.621086] Lustre: sultan-MDT000b: Connection restored to c60f26de-f3d6-6cfc-b87c-120a7b1f0eed (at 26@gni1) Aug 25 18:53:04 feral28.ccs.ornl.gov kernel: [1478246.631831] Lustre: Skipped 2 previous similar messages Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.440490] INFO: task crond:17243 blocked for more than 120 seconds. Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.447532] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.456845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.465442] crond D 0000000000000003 0 17243 11135 0x00000000 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.473018] ffff880bbe307d38 0000000000000082 0000000000000000 0000000000000000 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.481228] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.489406] 0000000000000000 0000000000000000 ffff880bd810c5f8 ffff880bbe307fd8 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.497593] Call Trace: Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.500447] [] schedule_timeout+0x215/0x2e0 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.506857] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.513389] [] wait_for_common+0x123/0x180 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.519676] [] ? default_wake_function+0x0/0x20 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.526338] [] wait_for_completion+0x1d/0x20 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.532812] [] synchronize_sched+0x58/0x60 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.539102] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.545425] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.552954] [] join_session_keyring+0x133/0x160 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.559663] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.566839] [] sys_keyctl+0x180/0x1a0 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.572672] [] system_call_fastpath+0x16/0x1b Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.579241] INFO: task crond:17344 blocked for more than 120 seconds. Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.586296] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.595489] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.604164] crond D 0000000000000002 0 17344 11135 0x00000000 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.611715] ffff880bb8d0fd38 0000000000000086 0000000000000000 0000000000000000 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.619865] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.628048] 0000000000000000 0000000000000000 ffff880bba1105f8 ffff880bb8d0ffd8 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.636248] Call Trace: Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.639087] [] schedule_timeout+0x215/0x2e0 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.645520] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.652008] [] wait_for_common+0x123/0x180 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.658310] [] ? default_wake_function+0x0/0x20 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.665044] [] wait_for_completion+0x1d/0x20 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.671510] [] synchronize_sched+0x58/0x60 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.677809] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.684049] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.691548] [] join_session_keyring+0x133/0x160 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.698241] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.705431] [] sys_keyctl+0x180/0x1a0 Aug 25 18:53:37 feral28.ccs.ornl.gov kernel: [1478279.711264] [] system_call_fastpath+0x16/0x1b Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.685972] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.688972] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.688973] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689015] CPU 5 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689016] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689053] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689055] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689058] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689062] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689064] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689065] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689067] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689069] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689070] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689072] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689074] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689076] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689078] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689080] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689082] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689083] Stack: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689084] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689086] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689088] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689090] Call Trace: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689134] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689180] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689230] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689277] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689323] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689339] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689350] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689362] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689373] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689388] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689404] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689433] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689448] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689463] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689513] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689517] [] ? strlcpy+0x4a/0x60 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689534] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689548] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689563] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689578] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689622] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689662] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689717] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689771] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689824] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689876] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689880] [] ? thread_return+0x4e/0x7d0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689932] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689935] [] ? kthread+0x9e/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689938] [] ? child_rip+0xa/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689941] [] ? kthread+0x0/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689943] [] ? child_rip+0x0/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689944] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.689959] Call Trace: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690002] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690049] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690098] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690145] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690191] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690207] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690217] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690229] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690240] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690255] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690271] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690299] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690314] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690329] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690379] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690383] [] ? strlcpy+0x4a/0x60 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690399] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690414] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690429] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690444] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690487] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690528] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690582] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690636] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690689] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690741] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690745] [] ? thread_return+0x4e/0x7d0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690796] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690800] [] ? kthread+0x9e/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690803] [] ? child_rip+0xa/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690806] [] ? kthread+0x0/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.690808] [] ? child_rip+0x0/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.692972] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.692974] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693016] CPU 6 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693017] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693054] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693056] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693059] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693063] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693065] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693067] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693068] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693070] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693072] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693074] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693076] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693077] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693079] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693081] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693083] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693084] Stack: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693085] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693087] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693089] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693092] Call Trace: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693127] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693173] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693223] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693270] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693317] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693332] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693343] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693355] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693367] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693417] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693428] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693443] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693474] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693491] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693519] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693534] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693549] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693599] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693603] [] ? strlcpy+0x4a/0x60 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693619] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693634] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693649] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693664] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693708] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693748] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693802] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693857] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693910] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693962] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.693965] [] ? thread_return+0x4e/0x7d0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694017] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694020] [] ? kthread+0x9e/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694023] [] ? child_rip+0xa/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694026] [] ? kthread+0x0/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694028] [] ? child_rip+0x0/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694029] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694045] Call Trace: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694087] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694134] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694183] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694230] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694276] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694292] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694302] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694314] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694326] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694376] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694387] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694402] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694431] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694448] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694476] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694491] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694506] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694556] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694560] [] ? strlcpy+0x4a/0x60 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694576] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694591] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694605] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694620] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694664] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694704] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694758] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694812] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694866] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694917] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694921] [] ? thread_return+0x4e/0x7d0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694973] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694976] [] ? kthread+0x9e/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694979] [] ? child_rip+0xa/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694981] [] ? kthread+0x0/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.694984] [] ? child_rip+0x0/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.695971] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.695972] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696013] CPU 7 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696014] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696050] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696051] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696054] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696058] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696060] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696062] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696063] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696065] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696066] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696068] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696070] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696072] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696075] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696077] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696078] Stack: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696079] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696081] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696083] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696086] Call Trace: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696120] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696166] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696216] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696263] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696309] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696325] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696335] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696348] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696358] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696373] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696390] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696418] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696433] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696448] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696498] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696502] [] ? strlcpy+0x4a/0x60 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696518] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696533] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696548] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696563] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696607] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696647] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696701] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696755] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696809] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696861] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696864] [] ? thread_return+0x4e/0x7d0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696916] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696919] [] ? kthread+0x9e/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696922] [] ? child_rip+0xa/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696925] [] ? kthread+0x0/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696927] [] ? child_rip+0x0/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696928] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696944] Call Trace: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.696986] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697033] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697082] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697129] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697175] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697191] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697201] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697213] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697224] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697239] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697255] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697283] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697298] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697313] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697363] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697367] [] ? strlcpy+0x4a/0x60 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697383] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697397] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697412] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697427] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697471] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697511] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697565] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697619] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697673] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697725] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697728] [] ? thread_return+0x4e/0x7d0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697780] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697783] [] ? kthread+0x9e/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697786] [] ? child_rip+0xa/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697789] [] ? kthread+0x0/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478301.697791] [] ? child_rip+0x0/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.200655] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.297664] CPU 4 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.299779] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.397091] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.398938] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.412353] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.420970] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.426811] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.434646] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.442498] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.450366] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.458229] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.466088] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.474989] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.481291] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.489116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.496969] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.504855] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.514276] Stack: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.516678] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.524531] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.532999] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.541829] Call Trace: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.544748] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.552120] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.560123] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.567702] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.575173] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.582690] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.590322] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.597630] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.604701] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.611611] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.619440] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.626638] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.634875] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.641176] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.648214] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.656123] [] ? strlcpy+0x4a/0x60 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.661745] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.668563] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.675982] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.683106] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.690388] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.698022] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.706035] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.713463] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.720465] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.728350] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.735443] [] ? thread_return+0x4e/0x7d0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.741741] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.748652] [] ? kthread+0x9e/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.754243] [] ? child_rip+0xa/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.759910] [] ? kthread+0x0/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.765379] [] ? child_rip+0x0/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.771019] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.792565] Call Trace: Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.795459] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.802795] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.810789] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.818463] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.825917] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.833410] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.840980] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.848305] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.855385] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.862210] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.869985] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.877129] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.885402] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.891711] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.898732] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.906591] [] ? strlcpy+0x4a/0x60 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.912263] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.919091] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.926484] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.933660] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.940961] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.948506] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.956563] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.963946] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.970933] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.978825] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.985876] [] ? thread_return+0x4e/0x7d0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.992138] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478304.999042] [] ? kthread+0x9e/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478305.004595] [] ? child_rip+0xa/0x20 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478305.010282] [] ? kthread+0x0/0xc0 Aug 25 18:54:03 feral28.ccs.ornl.gov kernel: [1478305.015811] [] ? child_rip+0x0/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.692590] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695590] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695592] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695634] CPU 5 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695635] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695671] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695673] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695676] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695680] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695681] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695683] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695685] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695686] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695688] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695690] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695692] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695693] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695695] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695697] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695699] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695700] Stack: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695701] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695703] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695705] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695708] Call Trace: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695752] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695798] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695848] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695894] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695941] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695957] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695967] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695979] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.695990] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696005] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696021] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696050] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696065] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696080] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696131] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696135] [] ? strlcpy+0x4a/0x60 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696151] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696166] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696180] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696195] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696239] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696280] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696334] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696388] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696442] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696494] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696497] [] ? thread_return+0x4e/0x7d0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696549] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696552] [] ? kthread+0x9e/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696555] [] ? child_rip+0xa/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696558] [] ? kthread+0x0/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696560] [] ? child_rip+0x0/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696561] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696577] Call Trace: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696619] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696666] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696715] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696762] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696809] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696824] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696835] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696847] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696857] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696872] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696888] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696917] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696932] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696947] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.696997] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697001] [] ? strlcpy+0x4a/0x60 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697017] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697032] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697046] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697061] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697105] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697146] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697200] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697254] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697307] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697359] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697363] [] ? thread_return+0x4e/0x7d0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697415] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697418] [] ? kthread+0x9e/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697421] [] ? child_rip+0xa/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697424] [] ? kthread+0x0/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.697426] [] ? child_rip+0x0/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699590] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699592] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699635] CPU 6 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699636] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699672] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699674] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699677] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699682] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699683] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699685] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699687] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699688] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699690] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699692] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699694] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699696] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699700] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699701] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699703] Stack: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699704] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699706] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699708] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699710] Call Trace: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699745] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699792] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699841] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699888] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699935] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699951] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699961] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699973] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.699985] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700036] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700046] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700061] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700090] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700107] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700136] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700150] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700165] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700216] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700220] [] ? strlcpy+0x4a/0x60 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700236] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700251] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700266] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700280] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700324] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700365] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700419] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700473] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700526] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700578] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700582] [] ? thread_return+0x4e/0x7d0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700634] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700637] [] ? kthread+0x9e/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700640] [] ? child_rip+0xa/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700643] [] ? kthread+0x0/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700645] [] ? child_rip+0x0/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700646] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700661] Call Trace: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700704] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700750] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700800] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700847] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700893] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700909] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700919] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700931] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700943] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.700993] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701004] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701019] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701050] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701066] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701095] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701109] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701124] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701175] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701179] [] ? strlcpy+0x4a/0x60 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701195] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701209] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701224] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701239] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701283] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701323] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701377] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701431] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701485] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701536] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701540] [] ? thread_return+0x4e/0x7d0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701592] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701595] [] ? kthread+0x9e/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701598] [] ? child_rip+0xa/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701601] [] ? kthread+0x0/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.701603] [] ? child_rip+0x0/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702590] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702591] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702631] CPU 7 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702632] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702668] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702670] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702673] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702677] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702678] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702680] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702681] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702683] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702684] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702686] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702688] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702690] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702693] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702695] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702697] Stack: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702697] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702700] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702702] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702704] Call Trace: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702738] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702785] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702835] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702881] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702928] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702943] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702954] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702966] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702976] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.702992] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703008] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703037] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703051] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703066] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703117] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703121] [] ? strlcpy+0x4a/0x60 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703137] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703152] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703166] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703181] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703225] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703266] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703320] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703374] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703427] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703479] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703483] [] ? thread_return+0x4e/0x7d0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703534] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703538] [] ? kthread+0x9e/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703541] [] ? child_rip+0xa/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703543] [] ? kthread+0x0/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703546] [] ? child_rip+0x0/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703547] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703562] Call Trace: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703604] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703651] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703700] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703747] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703794] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703809] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703820] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703832] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703842] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703858] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703874] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703902] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703917] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703932] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703982] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.703986] [] ? strlcpy+0x4a/0x60 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704002] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704017] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704031] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704046] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704090] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704130] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704184] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704238] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704292] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704343] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704347] [] ? thread_return+0x4e/0x7d0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704399] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704402] [] ? kthread+0x9e/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704405] [] ? child_rip+0xa/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704408] [] ? kthread+0x0/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478385.704410] [] ? child_rip+0x0/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.223304] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.320599] CPU 4 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.322742] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.420220] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.422083] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.435437] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.444114] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.450016] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.457921] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.465842] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.473745] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.481668] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.489615] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.498578] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.504896] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.512826] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.520747] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.528644] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.538090] Stack: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.540534] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.548404] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.556956] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.565790] Call Trace: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.568666] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.576060] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.584194] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.591921] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.599402] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.606998] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.614612] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.621953] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.629043] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.635952] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.643807] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.651022] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.659328] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.665742] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.672796] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.680725] [] ? strlcpy+0x4a/0x60 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.686371] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.693164] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.700583] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.707760] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.715078] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.722649] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.730706] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.738185] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.745136] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.752995] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.760071] [] ? thread_return+0x4e/0x7d0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.766377] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.773244] [] ? kthread+0x9e/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.778869] [] ? child_rip+0xa/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.784544] [] ? kthread+0x0/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.790110] [] ? child_rip+0x0/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.795785] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.817251] Call Trace: Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.820173] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.827586] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.835649] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.843343] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.850805] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.858358] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.866041] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.873400] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.880506] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.887443] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.895253] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.902407] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.910714] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.919798] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.926879] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.934789] [] ? strlcpy+0x4a/0x60 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.940383] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.947280] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.954637] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.961832] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.969124] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.976687] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.984822] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.992145] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478388.999085] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478389.006963] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478389.014083] [] ? thread_return+0x4e/0x7d0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478389.020354] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478389.027230] [] ? kthread+0x9e/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478389.032828] [] ? child_rip+0xa/0x20 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478389.038541] [] ? kthread+0x0/0xc0 Aug 25 18:55:27 feral28.ccs.ornl.gov kernel: [1478389.044086] [] ? child_rip+0x0/0x20 Aug 25 18:55:37 feral28.ccs.ornl.gov kernel: [1478399.726971] INFO: task crond:17243 blocked for more than 120 seconds. Aug 25 18:55:37 feral28.ccs.ornl.gov kernel: [1478399.734114] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:55:37 feral28.ccs.ornl.gov kernel: [1478399.743365] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:55:37 feral28.ccs.ornl.gov kernel: [1478399.751988] crond D 0000000000000003 0 17243 11135 0x00000000 Aug 25 18:55:37 feral28.ccs.ornl.gov kernel: [1478399.759467] ffff880bbe307d38 0000000000000082 0000000000000000 0000000000000000 Aug 25 18:55:37 feral28.ccs.ornl.gov kernel: [1478399.767653] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.775857] 0000000000000000 0000000000000000 ffff880bd810c5f8 ffff880bbe307fd8 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.784088] Call Trace: Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.786949] [] schedule_timeout+0x215/0x2e0 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.793363] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.799798] [] wait_for_common+0x123/0x180 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.806041] [] ? default_wake_function+0x0/0x20 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.812728] [] wait_for_completion+0x1d/0x20 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.819168] [] synchronize_sched+0x58/0x60 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.825468] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.831720] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.839258] [] join_session_keyring+0x133/0x160 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.845975] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.853142] [] sys_keyctl+0x180/0x1a0 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.859072] [] system_call_fastpath+0x16/0x1b Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.865630] INFO: task crond:17344 blocked for more than 120 seconds. Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.872668] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.881941] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.890571] crond D 0000000000000002 0 17344 11135 0x00000000 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.898140] ffff880bb8d0fd38 0000000000000086 0000000000000000 0000000000000000 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.906361] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.914552] 0000000000000000 0000000000000000 ffff880bba1105f8 ffff880bb8d0ffd8 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.922749] Call Trace: Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.925590] [] schedule_timeout+0x215/0x2e0 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.931950] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.938426] [] wait_for_common+0x123/0x180 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.944735] [] ? default_wake_function+0x0/0x20 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.951452] [] wait_for_completion+0x1d/0x20 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.957900] [] synchronize_sched+0x58/0x60 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.964140] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.970433] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.977972] [] join_session_keyring+0x133/0x160 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.984715] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.991909] [] sys_keyctl+0x180/0x1a0 Aug 25 18:55:38 feral28.ccs.ornl.gov kernel: [1478399.997726] [] system_call_fastpath+0x16/0x1b Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.699209] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702209] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702210] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702252] CPU 5 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702253] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702289] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702291] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702293] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702298] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702299] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702301] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702302] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702304] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702306] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702307] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702309] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702311] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702314] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702316] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702317] Stack: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702318] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702320] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702323] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702325] Call Trace: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702369] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702415] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702465] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702512] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702559] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702574] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702585] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702597] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702608] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702623] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702639] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702669] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702683] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702698] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702749] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702753] [] ? strlcpy+0x4a/0x60 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702769] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702784] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702799] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702814] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702858] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702898] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.702952] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703007] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703060] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703112] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703116] [] ? thread_return+0x4e/0x7d0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703168] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703171] [] ? kthread+0x9e/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703174] [] ? child_rip+0xa/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703176] [] ? kthread+0x0/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703179] [] ? child_rip+0x0/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703180] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703195] Call Trace: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703238] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703284] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703334] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703381] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703427] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703443] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703453] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703465] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703476] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703491] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703507] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703536] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703550] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703565] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703615] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703619] [] ? strlcpy+0x4a/0x60 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703636] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703650] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703665] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703680] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703724] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703764] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703818] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703872] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703926] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703977] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.703981] [] ? thread_return+0x4e/0x7d0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.704033] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.704036] [] ? kthread+0x9e/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.704039] [] ? child_rip+0xa/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.704042] [] ? kthread+0x0/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.704044] [] ? child_rip+0x0/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706209] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706211] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706254] CPU 6 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706255] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706292] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706293] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706296] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706301] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706302] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706304] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706305] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706307] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706309] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706311] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706313] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706315] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706316] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706318] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706320] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706322] Stack: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706323] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706325] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706327] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706330] Call Trace: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706364] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706411] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706461] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706507] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706554] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706570] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706581] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706592] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706605] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706655] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706666] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706681] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706712] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706729] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706757] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706772] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706787] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706837] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706841] [] ? strlcpy+0x4a/0x60 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706857] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706872] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706887] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706902] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706945] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.706986] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707040] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707094] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707148] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707199] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707203] [] ? thread_return+0x4e/0x7d0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707255] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707258] [] ? kthread+0x9e/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707261] [] ? child_rip+0xa/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707264] [] ? kthread+0x0/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707266] [] ? child_rip+0x0/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707267] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707282] Call Trace: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707325] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707371] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707421] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707467] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707514] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707529] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707540] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707552] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707564] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707614] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707625] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707640] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707669] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707686] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707714] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707729] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707744] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707795] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707799] [] ? strlcpy+0x4a/0x60 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707815] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707830] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707845] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707859] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707903] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707943] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.707997] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.708051] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.708104] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.708156] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.708160] [] ? thread_return+0x4e/0x7d0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.708212] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.708215] [] ? kthread+0x9e/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.708218] [] ? child_rip+0xa/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.708221] [] ? kthread+0x0/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.708223] [] ? child_rip+0x0/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709208] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709209] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709251] CPU 7 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709252] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709288] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709290] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709293] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709297] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709299] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709301] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709302] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709304] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709306] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709308] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709310] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709311] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709314] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709316] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709318] Stack: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709318] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709320] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709322] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709325] Call Trace: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709359] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709406] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709455] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709502] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709548] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709564] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709575] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709587] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709597] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709612] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709628] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709657] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709672] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709687] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709737] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709741] [] ? strlcpy+0x4a/0x60 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709757] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709772] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709787] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709802] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709846] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709886] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709940] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.709994] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710047] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710099] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710103] [] ? thread_return+0x4e/0x7d0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710154] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710158] [] ? kthread+0x9e/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710161] [] ? child_rip+0xa/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710164] [] ? kthread+0x0/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710166] [] ? child_rip+0x0/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710167] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710182] Call Trace: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710225] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710271] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710321] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710367] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710414] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710429] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710440] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710452] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710462] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710477] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710493] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710522] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710537] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710552] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710602] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710605] [] ? strlcpy+0x4a/0x60 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710622] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710636] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710651] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710666] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710709] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710750] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710804] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710858] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710911] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710963] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.710967] [] ? thread_return+0x4e/0x7d0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.711018] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.711022] [] ? kthread+0x9e/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.711025] [] ? child_rip+0xa/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.711027] [] ? kthread+0x0/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478469.711030] [] ? child_rip+0x0/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.210440] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.307336] CPU 4 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.309425] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.407301] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.409166] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.425224] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.433848] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.439699] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.447645] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.455550] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.463419] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.471270] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.479175] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.488035] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.494344] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.502212] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.510135] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.518022] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.527434] Stack: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.529843] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.537696] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.546144] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.554968] Call Trace: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.557897] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.565266] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.573235] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.580868] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.588295] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.595820] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.603435] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.610752] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.617771] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.624630] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.632440] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.639595] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.647797] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.654070] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.661091] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.668932] [] ? strlcpy+0x4a/0x60 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.674615] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.681416] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.688817] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.695985] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.703269] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.710892] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.718915] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.726264] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.733206] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.741091] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.748149] [] ? thread_return+0x4e/0x7d0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.754472] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.761332] [] ? kthread+0x9e/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.766920] [] ? child_rip+0xa/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.772598] [] ? kthread+0x0/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.778109] [] ? child_rip+0x0/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.783777] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.805271] Call Trace: Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.808164] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.815473] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.823476] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.831109] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.838571] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.846038] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.853590] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.860932] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.867986] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.874890] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.882647] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.889776] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.898029] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.904349] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.911326] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.919175] [] ? strlcpy+0x4a/0x60 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.924752] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.931553] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.938938] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.946105] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.953413] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.960976] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.968937] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.976315] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.983231] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.991151] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478472.998244] [] ? thread_return+0x4e/0x7d0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478473.004541] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478473.011418] [] ? kthread+0x9e/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478473.016997] [] ? child_rip+0xa/0x20 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478473.022641] [] ? kthread+0x0/0xc0 Aug 25 18:56:51 feral28.ccs.ornl.gov kernel: [1478473.028152] [] ? child_rip+0x0/0x20 Aug 25 18:57:37 feral28.ccs.ornl.gov kernel: [1478519.695395] Lustre: sultan-MDT000b: Client 1f0727a3-d49c-044b-6def-11f5d586ab1d (at 26@gni1) reconnecting Aug 25 18:57:37 feral28.ccs.ornl.gov kernel: [1478519.705766] Lustre: Skipped 4 previous similar messages Aug 25 18:57:37 feral28.ccs.ornl.gov kernel: [1478519.711535] Lustre: sultan-MDT000b: Connection restored to c60f26de-f3d6-6cfc-b87c-120a7b1f0eed (at 26@gni1) Aug 25 18:57:37 feral28.ccs.ornl.gov kernel: [1478519.722253] Lustre: Skipped 5 previous similar messages Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.013447] INFO: task crond:17243 blocked for more than 120 seconds. Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.020485] Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.029669] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.038250] crond D 0000000000000003 0 17243 11135 0x00000000 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.045789] ffff880bbe307d38 0000000000000082 0000000000000000 0000000000000000 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.053941] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.062112] 0000000000000000 0000000000000000 ffff880bd810c5f8 ffff880bbe307fd8 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.070289] Call Trace: Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.073142] [] schedule_timeout+0x215/0x2e0 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.079526] [] ? extract_entropy+0x108/0x1f0 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.086057] [] wait_for_common+0x123/0x180 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.092368] [] ? default_wake_function+0x0/0x20 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.099146] [] wait_for_completion+0x1d/0x20 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.105629] [] synchronize_sched+0x58/0x60 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.111937] [] ? wakeme_after_rcu+0x0/0x20 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.118256] [] install_session_keyring_to_cred+0x6c/0xd0 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.125803] [] join_session_keyring+0x133/0x160 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.132513] [] keyctl_join_session_keyring+0x38/0x70 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.139654] [] sys_keyctl+0x180/0x1a0 Aug 25 18:57:38 feral28.ccs.ornl.gov kernel: [1478520.145487] [] system_call_fastpath+0x16/0x1b Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.705826] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708826] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708827] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708870] CPU 5 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708871] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708907] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708909] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708912] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708916] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708918] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708919] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708921] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708922] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708924] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708926] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708928] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708930] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708932] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708933] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708935] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708937] Stack: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708938] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708940] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708942] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708944] Call Trace: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.708988] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709034] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709084] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709130] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709177] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709193] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709204] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709216] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709226] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709242] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709258] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709287] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709302] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709317] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709367] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709371] [] ? strlcpy+0x4a/0x60 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709387] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709402] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709417] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709431] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709475] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709516] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709570] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709624] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709678] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709730] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709733] [] ? thread_return+0x4e/0x7d0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709785] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709788] [] ? kthread+0x9e/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709791] [] ? child_rip+0xa/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709794] [] ? kthread+0x0/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709796] [] ? child_rip+0x0/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709797] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709812] Call Trace: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709855] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709901] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709951] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.709998] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710044] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710060] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710070] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710082] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710093] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710108] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710124] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710153] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710167] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710182] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710232] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710236] [] ? strlcpy+0x4a/0x60 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710252] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710267] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710282] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710297] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710340] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710381] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710435] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710489] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710542] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710594] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710597] [] ? thread_return+0x4e/0x7d0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710649] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710652] [] ? kthread+0x9e/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710655] [] ? child_rip+0xa/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710658] [] ? kthread+0x0/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.710660] [] ? child_rip+0x0/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712826] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712827] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712870] CPU 6 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712871] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712908] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712910] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712913] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712918] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712920] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712921] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712923] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712925] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712926] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712928] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712930] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712932] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712934] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712936] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712938] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712939] Stack: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712940] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712942] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712944] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712946] Call Trace: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.712981] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713028] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713078] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713124] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713171] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713187] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713197] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713209] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713221] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713272] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713282] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713298] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713327] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713344] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713373] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713387] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713402] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713453] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713457] [] ? strlcpy+0x4a/0x60 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713473] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713488] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713503] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713517] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713561] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713602] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713656] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713710] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713763] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713815] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713819] [] ? thread_return+0x4e/0x7d0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713871] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713874] [] ? kthread+0x9e/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713877] [] ? child_rip+0xa/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713880] [] ? kthread+0x0/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713882] [] ? child_rip+0x0/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713883] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713898] Call Trace: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713941] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.713987] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714037] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714083] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714130] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714145] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714156] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714167] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714179] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714229] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714240] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714255] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714284] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714301] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714329] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714343] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714358] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714409] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714413] [] ? strlcpy+0x4a/0x60 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714429] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714443] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714458] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714473] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714517] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714557] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714611] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714665] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714719] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714771] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714774] [] ? thread_return+0x4e/0x7d0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714826] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714830] [] ? kthread+0x9e/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714832] [] ? child_rip+0xa/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714835] [] ? kthread+0x0/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.714837] [] ? child_rip+0x0/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715825] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715826] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715866] CPU 7 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715867] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715903] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715905] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715907] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715911] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715913] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715915] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715916] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715918] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715919] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715921] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715923] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715925] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715928] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715930] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715931] Stack: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715932] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715934] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715936] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715939] Call Trace: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.715973] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716019] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716069] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716116] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716162] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716178] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716188] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716201] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716211] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716226] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716242] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716271] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716286] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716301] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716352] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716355] [] ? strlcpy+0x4a/0x60 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716372] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716386] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716401] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716416] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716460] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716500] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716554] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716608] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716662] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716713] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716717] [] ? thread_return+0x4e/0x7d0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716769] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716772] [] ? kthread+0x9e/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716775] [] ? child_rip+0xa/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716778] [] ? kthread+0x0/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716780] [] ? child_rip+0x0/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716781] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716796] Call Trace: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716839] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716885] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716935] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.716981] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717028] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717044] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717054] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717066] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717077] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717092] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717108] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717137] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717151] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717166] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717217] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717220] [] ? strlcpy+0x4a/0x60 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717237] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717251] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717266] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717281] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717325] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717365] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717419] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717473] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717527] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717578] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717582] [] ? thread_return+0x4e/0x7d0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717634] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717637] [] ? kthread+0x9e/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717640] [] ? child_rip+0xa/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717643] [] ? kthread+0x0/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478553.717645] [] ? child_rip+0x0/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.236938] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.334546] CPU 4 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.336651] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.434303] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.436140] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.449583] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.458189] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.464056] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.471987] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.479935] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.487934] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.495854] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.503822] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.512705] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.519023] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.526927] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.534866] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.542744] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.552130] Stack: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.554513] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.562409] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.570927] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.579752] Call Trace: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.582636] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.589989] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.597992] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.605607] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.613078] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.620596] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.628209] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.635466] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.642484] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.649319] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.657145] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.664274] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.672484] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.678812] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.685807] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.693726] [] ? strlcpy+0x4a/0x60 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.699337] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.706148] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.713497] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.720672] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.728001] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.735579] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.743680] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.751063] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.758024] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.765960] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.773037] [] ? thread_return+0x4e/0x7d0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.779341] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.786245] [] ? kthread+0x9e/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.791851] [] ? child_rip+0xa/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.797545] [] ? kthread+0x0/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.803006] [] ? child_rip+0x0/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.808690] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.830200] Call Trace: Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.833076] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.840431] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.848442] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.856108] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.863520] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.871003] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.878589] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.885923] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.892941] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.899803] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.907621] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.914732] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.922969] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.929234] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.936212] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.944112] [] ? strlcpy+0x4a/0x60 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.949725] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.956537] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.963842] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.971001] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.978319] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.985864] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478556.994000] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478557.001453] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478557.008412] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478557.016289] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478557.023311] [] ? thread_return+0x4e/0x7d0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478557.029566] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478557.036496] [] ? kthread+0x9e/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478557.044746] [] ? child_rip+0xa/0x20 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478557.050431] [] ? kthread+0x0/0xc0 Aug 25 18:58:15 feral28.ccs.ornl.gov kernel: [1478557.055943] [] ? child_rip+0x0/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.712441] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715442] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715443] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715486] CPU 5 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715487] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715523] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715524] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715527] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715532] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715533] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715535] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715536] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715538] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715539] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715541] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715543] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715545] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715548] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715550] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715551] Stack: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715552] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715554] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715557] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715559] Call Trace: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715603] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715650] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715699] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715746] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715792] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715808] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715819] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715831] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715842] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715857] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715873] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715902] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715917] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715932] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715982] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.715986] [] ? strlcpy+0x4a/0x60 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716002] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716017] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716032] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716047] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716091] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716131] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716185] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716240] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716293] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716345] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716349] [] ? thread_return+0x4e/0x7d0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716401] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716404] [] ? kthread+0x9e/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716407] [] ? child_rip+0xa/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716410] [] ? kthread+0x0/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716412] [] ? child_rip+0x0/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716413] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716428] Call Trace: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716471] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716517] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716567] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716613] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716660] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716676] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716686] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716698] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716709] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716724] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716740] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716769] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716783] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716798] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716848] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716852] [] ? strlcpy+0x4a/0x60 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716868] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716883] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716898] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716913] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716956] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.716997] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717051] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717105] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717158] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717210] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717214] [] ? thread_return+0x4e/0x7d0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717266] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717269] [] ? kthread+0x9e/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717272] [] ? child_rip+0xa/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717275] [] ? kthread+0x0/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.717277] [] ? child_rip+0x0/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719441] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719443] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719486] CPU 6 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719487] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719523] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719525] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719528] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719532] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719534] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719536] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719537] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719539] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719541] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719543] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719545] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719547] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719550] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719552] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719554] Stack: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719555] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719556] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719559] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719561] Call Trace: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719596] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719642] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719692] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719739] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719785] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719801] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719812] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719823] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719836] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719886] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719897] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719912] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719943] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719959] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.719988] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720003] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720018] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720068] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720072] [] ? strlcpy+0x4a/0x60 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720088] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720103] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720118] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720133] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720176] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720217] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720271] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720325] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720379] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720431] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720434] [] ? thread_return+0x4e/0x7d0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720486] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720489] [] ? kthread+0x9e/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720492] [] ? child_rip+0xa/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720495] [] ? kthread+0x0/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720497] [] ? child_rip+0x0/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720499] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720514] Call Trace: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720556] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720603] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720653] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720699] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720746] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720761] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720772] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720783] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720795] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720845] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720856] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720871] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720902] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720919] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720947] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720962] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.720977] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721027] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721031] [] ? strlcpy+0x4a/0x60 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721047] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721061] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721076] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721091] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721135] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721176] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721230] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721283] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721337] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721389] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721392] [] ? thread_return+0x4e/0x7d0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721444] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721448] [] ? kthread+0x9e/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721451] [] ? child_rip+0xa/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721453] [] ? kthread+0x0/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.721456] [] ? child_rip+0x0/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722441] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722442] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722482] CPU 7 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722483] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722519] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722521] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722524] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722528] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722530] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722531] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722533] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722534] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722536] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722538] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722540] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722541] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722543] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722545] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722546] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722548] Stack: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722549] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722550] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722553] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722555] Call Trace: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722589] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722635] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722685] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722731] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722778] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722794] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722804] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722816] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722827] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722842] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722858] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722887] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722901] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722916] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722967] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722971] [] ? strlcpy+0x4a/0x60 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.722987] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723002] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723017] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723032] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723075] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723116] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723170] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723224] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723277] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723329] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723332] [] ? thread_return+0x4e/0x7d0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723384] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723387] [] ? kthread+0x9e/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723390] [] ? child_rip+0xa/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723393] [] ? kthread+0x0/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723395] [] ? child_rip+0x0/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723396] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723411] Call Trace: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723454] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723500] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723550] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723596] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723643] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723658] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723669] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723681] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723691] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723707] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723723] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723751] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723766] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723781] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723831] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723835] [] ? strlcpy+0x4a/0x60 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723851] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723866] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723881] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723896] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723939] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.723980] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724034] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724087] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724141] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724193] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724196] [] ? thread_return+0x4e/0x7d0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724248] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724251] [] ? kthread+0x9e/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724254] [] ? child_rip+0xa/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724257] [] ? kthread+0x0/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478637.724259] [] ? child_rip+0x0/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.239673] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.336950] CPU 4 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.339049] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.436473] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.438328] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.451737] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.460369] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.466262] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.474149] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.482053] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.490002] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.497915] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.505889] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.514824] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.521185] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.529150] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.537045] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.544967] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.554412] Stack: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.556814] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.564702] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.573273] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.582129] Call Trace: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.585041] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.592385] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.600439] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.608098] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.615605] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.623123] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.630779] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.638199] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.645323] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.652208] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.660070] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.667267] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.675582] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.681936] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.688949] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.696875] [] ? strlcpy+0x4a/0x60 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.702530] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.709375] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.716742] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.723971] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.731202] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.738738] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.746795] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.754249] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.761131] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.769041] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.776065] [] ? thread_return+0x4e/0x7d0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.782387] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.789238] [] ? kthread+0x9e/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.794820] [] ? child_rip+0xa/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.800523] [] ? kthread+0x0/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.808670] [] ? child_rip+0x0/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.814397] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.835883] Call Trace: Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.838785] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.846206] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.854288] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.861972] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.869443] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.877030] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.884635] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.891969] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.899030] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.905961] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.913839] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.921029] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.929266] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.935670] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.942665] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.950601] [] ? strlcpy+0x4a/0x60 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.956213] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.963075] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.970450] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.977671] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.984970] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478640.992603] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.000668] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.008086] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.014995] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.022888] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.030008] [] ? thread_return+0x4e/0x7d0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.036253] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.043112] [] ? kthread+0x9e/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.048728] [] ? child_rip+0xa/0x20 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.054448] [] ? kthread+0x0/0xc0 Aug 25 18:59:39 feral28.ccs.ornl.gov kernel: [1478641.060046] [] ? child_rip+0x0/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.719057] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722058] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722059] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722101] CPU 5 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722102] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722139] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722140] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722143] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722148] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722149] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722151] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722152] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722154] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722156] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722158] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722160] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722161] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722165] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722167] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722168] Stack: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722169] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722171] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722174] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722176] Call Trace: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722220] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722266] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722316] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722363] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722410] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722425] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722436] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722448] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722459] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722474] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722490] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722519] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722534] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722549] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722599] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722603] [] ? strlcpy+0x4a/0x60 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722619] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722634] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722649] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722664] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722707] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722748] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722802] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722856] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722910] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722962] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.722965] [] ? thread_return+0x4e/0x7d0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723017] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723020] [] ? kthread+0x9e/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723023] [] ? child_rip+0xa/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723026] [] ? kthread+0x0/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723028] [] ? child_rip+0x0/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723029] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723045] Call Trace: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723087] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723134] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723183] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723230] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723277] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723292] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723303] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723315] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723325] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723340] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723356] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723385] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723400] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723415] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723465] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723469] [] ? strlcpy+0x4a/0x60 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723485] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723500] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723515] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723529] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723573] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723613] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723667] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723722] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723775] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723827] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723831] [] ? thread_return+0x4e/0x7d0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723882] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723886] [] ? kthread+0x9e/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723889] [] ? child_rip+0xa/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723892] [] ? kthread+0x0/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.723894] [] ? child_rip+0x0/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726057] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726059] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726102] CPU 6 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726103] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726138] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726140] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726143] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726148] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726149] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726151] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726152] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726154] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726156] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726158] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726160] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726162] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726165] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726167] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726169] Stack: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726170] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726172] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726174] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726176] Call Trace: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726211] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726258] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726308] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726354] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726401] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726416] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726427] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726439] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726451] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726502] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726512] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726528] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726557] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726574] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726603] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726617] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726632] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726683] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726687] [] ? strlcpy+0x4a/0x60 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726703] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726718] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726733] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726747] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726791] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726832] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726886] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726940] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.726993] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727045] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727049] [] ? thread_return+0x4e/0x7d0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727100] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727104] [] ? kthread+0x9e/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727107] [] ? child_rip+0xa/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727109] [] ? kthread+0x0/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727111] [] ? child_rip+0x0/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727113] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727128] Call Trace: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727171] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727217] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727267] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727313] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727360] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727375] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727386] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727398] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727410] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727460] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727470] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727486] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727516] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727533] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727561] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727576] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727591] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727641] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727645] [] ? strlcpy+0x4a/0x60 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727661] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727676] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727690] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727705] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727749] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727789] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727843] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727898] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.727951] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.728003] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.728007] [] ? thread_return+0x4e/0x7d0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.728058] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.728062] [] ? kthread+0x9e/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.728065] [] ? child_rip+0xa/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.728067] [] ? kthread+0x0/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.728070] [] ? child_rip+0x0/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729057] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729058] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729098] CPU 7 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729099] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729135] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729137] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729140] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729144] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729146] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729147] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729149] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729151] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729152] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729154] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729156] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729158] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729160] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729162] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729164] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729165] Stack: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729166] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729168] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729170] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729172] Call Trace: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729206] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729253] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729303] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729349] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729396] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729411] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729422] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729434] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729444] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729460] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729476] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729505] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729519] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729534] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729585] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729588] [] ? strlcpy+0x4a/0x60 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729605] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729619] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729634] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729649] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729693] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729733] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729787] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729841] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729895] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729947] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.729950] [] ? thread_return+0x4e/0x7d0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730001] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730005] [] ? kthread+0x9e/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730008] [] ? child_rip+0xa/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730011] [] ? kthread+0x0/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730013] [] ? child_rip+0x0/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730014] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730029] Call Trace: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730072] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730118] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730168] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730214] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730261] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730276] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730287] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730299] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730309] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730324] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730340] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730369] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730384] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730399] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730449] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730453] [] ? strlcpy+0x4a/0x60 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730469] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730484] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730498] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730513] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730557] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730597] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730651] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730705] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730758] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730810] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730814] [] ? thread_return+0x4e/0x7d0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730865] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730869] [] ? kthread+0x9e/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730871] [] ? child_rip+0xa/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730874] [] ? kthread+0x0/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478721.730876] [] ? child_rip+0x0/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.232102] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.329181] CPU 4 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.331287] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.428625] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.430472] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.443784] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.452425] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.458309] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.466187] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.474074] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.481962] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.489867] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.497737] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.506637] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.512938] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.520797] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.528669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.536608] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.546019] Stack: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.548395] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.556221] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.564671] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.576189] Call Trace: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.579084] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.586444] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.594464] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.602106] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.609594] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.617077] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.624673] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.632016] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.639086] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.645947] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.653766] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.660901] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.669157] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.675447] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.682453] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.690327] [] ? strlcpy+0x4a/0x60 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.696024] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.702914] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.710229] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.717448] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.724776] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.732364] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.740413] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.747847] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.754773] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.762650] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.769760] [] ? thread_return+0x4e/0x7d0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.776109] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.783003] [] ? kthread+0x9e/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.788619] [] ? child_rip+0xa/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.794253] [] ? kthread+0x0/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.799712] [] ? child_rip+0x0/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.805362] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.826838] Call Trace: Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.829723] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.837066] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.845078] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.852676] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.860121] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.867629] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.875192] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.882516] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.889579] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.896473] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.904301] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.911439] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.919666] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.926002] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.933039] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.940908] [] ? strlcpy+0x4a/0x60 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.946475] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.953328] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.960731] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.967907] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.975233] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.982813] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.990860] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478724.998229] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478725.005179] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478725.013064] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478725.020183] [] ? thread_return+0x4e/0x7d0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478725.026421] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478725.033340] [] ? kthread+0x9e/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478725.038913] [] ? child_rip+0xa/0x20 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478725.044615] [] ? kthread+0x0/0xc0 Aug 25 19:01:03 feral28.ccs.ornl.gov kernel: [1478725.050093] [] ? child_rip+0x0/0x20 Aug 25 19:01:18 feral28.ccs.ornl.gov kernel: [1478740.734758] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 19:01:18 feral28.ccs.ornl.gov kernel: [1478740.745819] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 33 previous similar messages Aug 25 19:01:18 feral28.ccs.ornl.gov kernel: [1478740.756968] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.83@o2ib1 (222): c: 63, oc: 0, rc: 63 Aug 25 19:01:18 feral28.ccs.ornl.gov kernel: [1478740.770320] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 33 previous similar messages Aug 25 19:02:20 feral28.ccs.ornl.gov kernel: [1478801.959255] Lustre: sultan-MDT000b: haven't heard from client 1561233c-aa4b-81d5-e0e4-de3b3609634b (at 2@gni1) in 339 seconds. I think it's dead, and I am evicting it. exp ffff880be901c800, cur 1472166140 expire 1472165918 last 1472165801 Aug 25 19:02:20 feral28.ccs.ornl.gov kernel: [1478801.981826] Lustre: Skipped 1 previous similar message Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.725673] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728673] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728675] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728716] CPU 5 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728717] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728753] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728754] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728757] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728761] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728763] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728765] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728766] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728768] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728769] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728771] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728773] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728775] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728776] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728778] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728780] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728781] Stack: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728782] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728784] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728787] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728789] Call Trace: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728833] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728879] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728929] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.728976] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729022] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729038] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729049] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729061] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729071] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729086] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729103] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729132] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729146] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729161] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729212] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729216] [] ? strlcpy+0x4a/0x60 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729232] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729247] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729262] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729277] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729321] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729361] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729415] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729469] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729523] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729575] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729578] [] ? thread_return+0x4e/0x7d0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729630] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729633] [] ? kthread+0x9e/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729636] [] ? child_rip+0xa/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729639] [] ? kthread+0x0/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729641] [] ? child_rip+0x0/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729642] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729658] Call Trace: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729700] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729746] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729796] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729843] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729889] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729905] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729915] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729927] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729938] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729953] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729969] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.729998] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730012] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730027] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730077] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730081] [] ? strlcpy+0x4a/0x60 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730097] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730112] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730127] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730142] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730185] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730226] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730280] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730334] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730387] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730439] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730443] [] ? thread_return+0x4e/0x7d0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730495] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730498] [] ? kthread+0x9e/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730501] [] ? child_rip+0xa/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730504] [] ? kthread+0x0/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.730506] [] ? child_rip+0x0/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732673] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732675] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732717] CPU 6 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732718] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732755] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732757] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732760] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732764] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732766] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732768] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732769] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732771] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732773] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732775] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732777] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732778] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732780] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732782] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732784] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732785] Stack: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732786] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732788] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732790] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732793] Call Trace: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732828] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732874] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732924] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.732971] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733017] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733033] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733044] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733055] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733068] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733118] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733128] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733144] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733175] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733191] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733220] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733235] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733250] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733300] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733304] [] ? strlcpy+0x4a/0x60 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733320] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733335] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733350] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733365] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733408] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733449] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733503] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733557] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733611] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733663] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733666] [] ? thread_return+0x4e/0x7d0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733718] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733721] [] ? kthread+0x9e/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733724] [] ? child_rip+0xa/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733727] [] ? kthread+0x0/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733729] [] ? child_rip+0x0/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733730] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733745] Call Trace: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733788] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733835] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733884] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733931] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733977] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.733993] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734003] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734015] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734027] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734077] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734088] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734103] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734134] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734151] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734179] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734194] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734209] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734259] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734263] [] ? strlcpy+0x4a/0x60 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734279] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734294] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734309] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734324] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734368] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734408] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734462] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734516] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734569] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734621] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734625] [] ? thread_return+0x4e/0x7d0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734677] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734680] [] ? kthread+0x9e/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734683] [] ? child_rip+0xa/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734686] [] ? kthread+0x0/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.734688] [] ? child_rip+0x0/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735672] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735674] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735714] CPU 7 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735714] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735751] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735753] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735755] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735760] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735761] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735763] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735764] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735766] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735767] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735769] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735771] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735773] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735776] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735778] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735780] Stack: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735780] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735782] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735785] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735787] Call Trace: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735821] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735868] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735917] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.735964] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736010] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736026] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736037] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736049] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736059] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736074] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736090] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736119] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736134] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736149] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736199] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736203] [] ? strlcpy+0x4a/0x60 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736220] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736234] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736249] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736264] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736308] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736349] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736403] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736456] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736510] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736562] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736565] [] ? thread_return+0x4e/0x7d0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736617] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736621] [] ? kthread+0x9e/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736624] [] ? child_rip+0xa/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736626] [] ? kthread+0x0/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736629] [] ? child_rip+0x0/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736630] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736645] Call Trace: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736687] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736734] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736783] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736830] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736876] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736892] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736903] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736915] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736925] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736940] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736956] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.736985] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737000] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737015] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737065] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737069] [] ? strlcpy+0x4a/0x60 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737085] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737100] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737115] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737129] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737173] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737213] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737267] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737321] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737375] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737427] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737430] [] ? thread_return+0x4e/0x7d0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737482] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737485] [] ? kthread+0x9e/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737488] [] ? child_rip+0xa/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737491] [] ? kthread+0x0/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478805.737493] [] ? child_rip+0x0/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.254874] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.355039] CPU 4 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.357127] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.455056] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.456885] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.470344] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.479020] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.484861] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.492731] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.500653] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.508574] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.516530] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.524478] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.533466] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.539793] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.547706] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.555627] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.563593] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.573100] Stack: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.575509] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.583404] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.591863] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.600643] Call Trace: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.603580] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.610933] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.619022] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.626705] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.634228] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.641773] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.649324] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.656652] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.663782] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.670652] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.678505] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.685669] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.693973] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.700386] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.707470] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.715388] [] ? strlcpy+0x4a/0x60 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.720990] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.727854] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.735194] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.742371] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.749654] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.757259] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.765351] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.772787] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.779773] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.787641] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.794743] [] ? thread_return+0x4e/0x7d0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.801040] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.807959] [] ? kthread+0x9e/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.813540] [] ? child_rip+0xa/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.819252] [] ? kthread+0x0/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.824727] [] ? child_rip+0x0/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.830414] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.851864] Call Trace: Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.854800] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.862189] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.870313] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.878014] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.885476] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.893003] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.900599] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.907942] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.915055] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.921916] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.929769] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.936889] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.945247] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.951644] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.958682] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.966626] [] ? strlcpy+0x4a/0x60 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.972229] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.979066] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.986422] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478808.993598] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.000815] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.008325] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.016503] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.023939] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.030845] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.038783] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.045859] [] ? thread_return+0x4e/0x7d0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.052157] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.059042] [] ? kthread+0x9e/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.064657] [] ? child_rip+0xa/0x20 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.070325] [] ? kthread+0x0/0xc0 Aug 25 19:02:27 feral28.ccs.ornl.gov kernel: [1478809.075854] [] ? child_rip+0x0/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.732289] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735289] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735291] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735333] CPU 5 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735334] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735370] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735371] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735374] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735378] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735380] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735382] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735383] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735385] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735386] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735388] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735390] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735392] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735395] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735397] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735398] Stack: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735399] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735401] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735403] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735406] Call Trace: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735450] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735496] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735546] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735593] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735639] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735655] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735666] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735678] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735688] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735703] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735720] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735749] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735763] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735778] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735829] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735833] [] ? strlcpy+0x4a/0x60 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735849] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735864] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735879] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735894] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735937] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.735978] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736032] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736086] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736140] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736192] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736195] [] ? thread_return+0x4e/0x7d0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736247] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736250] [] ? kthread+0x9e/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736253] [] ? child_rip+0xa/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736256] [] ? kthread+0x0/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736258] [] ? child_rip+0x0/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736259] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736274] Call Trace: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736317] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736363] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736413] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736459] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736506] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736521] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736532] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736544] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736554] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736569] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736586] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736614] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736629] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736644] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736694] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736698] [] ? strlcpy+0x4a/0x60 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736714] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736729] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736744] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736759] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736803] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736843] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736897] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.736951] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.737004] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.737056] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.737059] [] ? thread_return+0x4e/0x7d0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.737111] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.737115] [] ? kthread+0x9e/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.737117] [] ? child_rip+0xa/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.737120] [] ? kthread+0x0/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.737122] [] ? child_rip+0x0/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739289] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739291] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739334] CPU 6 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739335] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739372] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739374] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739377] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739381] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739383] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739384] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739386] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739388] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739389] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739391] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739393] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739395] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739397] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739399] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739401] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739402] Stack: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739403] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739405] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739407] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739410] Call Trace: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739444] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739491] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739541] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739587] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739634] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739650] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739660] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739672] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739684] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739734] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739745] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739761] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739792] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739808] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739837] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739852] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739867] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739917] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739921] [] ? strlcpy+0x4a/0x60 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739937] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739952] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739967] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.739982] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740026] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740066] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740120] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740174] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740227] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740279] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740283] [] ? thread_return+0x4e/0x7d0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740335] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740338] [] ? kthread+0x9e/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740341] [] ? child_rip+0xa/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740344] [] ? kthread+0x0/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740346] [] ? child_rip+0x0/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740347] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740362] Call Trace: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740405] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740451] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740501] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740548] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740594] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740610] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740620] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740632] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740644] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740694] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740705] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740720] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740749] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740765] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740794] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740808] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740823] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740873] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740877] [] ? strlcpy+0x4a/0x60 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740893] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740908] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740923] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740938] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.740982] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741022] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741076] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741130] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741184] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741236] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741239] [] ? thread_return+0x4e/0x7d0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741291] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741294] [] ? kthread+0x9e/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741297] [] ? child_rip+0xa/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741300] [] ? kthread+0x0/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.741302] [] ? child_rip+0x0/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742288] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742290] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742329] CPU 7 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742330] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742367] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742368] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742371] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742376] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742377] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742379] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742381] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742382] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742384] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742386] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742388] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742390] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742392] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742393] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742395] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742396] Stack: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742397] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742399] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742401] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742404] Call Trace: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742438] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742484] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742534] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742580] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742627] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742643] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742653] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742665] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742676] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742691] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742707] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742736] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742751] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742766] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742816] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742820] [] ? strlcpy+0x4a/0x60 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742836] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742851] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742866] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742881] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742925] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.742965] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743019] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743073] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743127] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743179] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743182] [] ? thread_return+0x4e/0x7d0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743234] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743237] [] ? kthread+0x9e/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743240] [] ? child_rip+0xa/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743243] [] ? kthread+0x0/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743245] [] ? child_rip+0x0/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743246] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743262] Call Trace: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743304] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743351] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743400] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743447] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743494] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743509] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743520] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743532] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743542] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743557] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743573] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743602] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743617] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743632] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743682] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743686] [] ? strlcpy+0x4a/0x60 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743702] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743717] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743732] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743746] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743790] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743830] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743885] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743938] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.743992] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.744044] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.744047] [] ? thread_return+0x4e/0x7d0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.744099] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.744102] [] ? kthread+0x9e/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.744105] [] ? child_rip+0xa/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.744108] [] ? kthread+0x0/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478889.744110] [] ? child_rip+0x0/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.264257] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.361516] CPU 4 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.363666] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.461178] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.463009] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.476440] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.485117] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.490974] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.498863] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.506724] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.514619] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.522472] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.530358] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.539364] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.545708] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.553656] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.561595] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.569515] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.578988] Stack: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.581415] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.589389] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.597933] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.606801] Call Trace: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.609747] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.617160] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.625240] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.632838] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.640290] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.647844] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.655467] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.662895] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.669984] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.676878] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.684723] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.691886] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.700175] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.706441] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.713515] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.721451] [] ? strlcpy+0x4a/0x60 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.727095] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.733907] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.741300] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.748459] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.755777] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.763337] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.771448] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.778719] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.785687] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.793581] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.800700] [] ? thread_return+0x4e/0x7d0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.807024] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.813875] [] ? kthread+0x9e/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.819421] [] ? child_rip+0xa/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.825123] [] ? kthread+0x0/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.830661] [] ? child_rip+0x0/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.836353] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.857805] Call Trace: Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.860663] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.868043] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.876132] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.883825] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.891366] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.898838] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.906435] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.913872] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.920927] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.927857] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.935684] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.942847] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.951136] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.957489] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.964544] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.972454] [] ? strlcpy+0x4a/0x60 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.978075] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.984911] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.992322] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478892.999515] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.006868] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.014439] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.022470] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.029889] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.036805] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.044681] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.051765] [] ? thread_return+0x4e/0x7d0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.058124] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.065016] [] ? kthread+0x9e/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.070666] [] ? child_rip+0xa/0x20 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.076371] [] ? kthread+0x0/0xc0 Aug 25 19:03:51 feral28.ccs.ornl.gov kernel: [1478893.081882] [] ? child_rip+0x0/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.738905] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741905] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741906] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741948] CPU 5 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741949] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741984] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741986] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741989] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741993] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741995] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741996] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741998] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.741999] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742001] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742003] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742005] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742006] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742008] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742012] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742013] Stack: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742014] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742016] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742018] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742021] Call Trace: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742064] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742111] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742161] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742208] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742254] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742270] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742281] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742293] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742303] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742318] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742335] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742364] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742378] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742393] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742444] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742448] [] ? strlcpy+0x4a/0x60 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742464] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742479] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742493] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742508] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742552] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742592] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742646] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742700] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742754] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742806] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742809] [] ? thread_return+0x4e/0x7d0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742861] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742864] [] ? kthread+0x9e/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742867] [] ? child_rip+0xa/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742870] [] ? kthread+0x0/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742872] [] ? child_rip+0x0/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742873] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742888] Call Trace: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742931] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.742977] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743027] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743073] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743120] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743135] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743146] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743158] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743168] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743183] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743199] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743228] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743243] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743258] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743308] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743312] [] ? strlcpy+0x4a/0x60 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743328] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743343] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743358] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743373] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743417] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743457] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743511] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743565] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743618] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743670] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743674] [] ? thread_return+0x4e/0x7d0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743726] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743729] [] ? kthread+0x9e/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743732] [] ? child_rip+0xa/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743734] [] ? kthread+0x0/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.743736] [] ? child_rip+0x0/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.745905] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.745907] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.745949] CPU 6 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.745950] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.745987] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.745989] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.745992] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.745997] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.745998] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746000] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746001] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746003] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746005] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746007] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746009] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746011] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746014] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746016] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746018] Stack: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746019] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746021] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746023] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746026] Call Trace: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746060] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746107] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746157] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746203] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746250] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746266] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746277] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746288] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746301] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746351] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746361] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746377] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746406] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746423] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746451] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746466] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746481] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746531] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746535] [] ? strlcpy+0x4a/0x60 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746551] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746566] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746581] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746596] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746640] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746680] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746734] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746788] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746842] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746893] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746897] [] ? thread_return+0x4e/0x7d0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746949] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746952] [] ? kthread+0x9e/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746955] [] ? child_rip+0xa/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746958] [] ? kthread+0x0/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746960] [] ? child_rip+0x0/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746961] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.746976] Call Trace: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747019] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747065] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747115] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747162] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747208] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747224] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747235] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747246] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747258] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747309] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747319] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747334] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747363] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747379] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747408] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747423] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747438] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747488] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747492] [] ? strlcpy+0x4a/0x60 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747508] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747523] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747538] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747553] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747597] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747637] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747691] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747745] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747798] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747850] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747854] [] ? thread_return+0x4e/0x7d0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747906] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747909] [] ? kthread+0x9e/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747912] [] ? child_rip+0xa/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747914] [] ? kthread+0x0/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.747917] [] ? child_rip+0x0/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748904] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748906] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748945] CPU 7 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748946] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748982] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748984] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748987] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748991] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748993] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748994] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748996] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748998] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.748999] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749001] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749003] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749005] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749008] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749010] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749011] Stack: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749012] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749014] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749016] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749019] Call Trace: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749052] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749099] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749149] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749195] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749242] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749257] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749268] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749280] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749290] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749306] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749322] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749351] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749365] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749380] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749431] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749435] [] ? strlcpy+0x4a/0x60 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749451] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749465] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749480] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749495] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749539] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749579] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749633] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749687] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749741] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749793] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749796] [] ? thread_return+0x4e/0x7d0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749848] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749852] [] ? kthread+0x9e/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749855] [] ? child_rip+0xa/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749857] [] ? kthread+0x0/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749859] [] ? child_rip+0x0/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749861] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749876] Call Trace: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749918] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.749965] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750015] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750061] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750108] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750123] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750134] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750146] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750156] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750171] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750187] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750216] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750231] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750245] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750295] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750299] [] ? strlcpy+0x4a/0x60 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750316] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750330] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750345] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750360] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750404] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750444] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750498] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750552] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750606] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750657] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750661] [] ? thread_return+0x4e/0x7d0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750713] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750716] [] ? kthread+0x9e/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750719] [] ? child_rip+0xa/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750722] [] ? kthread+0x0/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478973.750724] [] ? child_rip+0x0/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.271279] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.368489] CPU 4 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.370561] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.468159] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.470049] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.483587] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.492306] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.498165] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.506086] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.513991] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.521980] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.529946] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.537912] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.546830] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.553182] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.561027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.568957] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.576862] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.586325] Stack: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.588735] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.596698] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.605236] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.614075] Call Trace: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.616979] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.624427] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.632499] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.640184] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.647663] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.655250] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.662889] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.670189] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.677216] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.684111] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.691955] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.699171] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.707459] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.713805] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.720755] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.728752] [] ? strlcpy+0x4a/0x60 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.734424] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.741139] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.748461] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.755656] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.763017] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.770615] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.778655] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.786047] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.792981] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.800900] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.807985] [] ? thread_return+0x4e/0x7d0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.814317] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.821237] [] ? kthread+0x9e/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.826844] [] ? child_rip+0xa/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.832572] [] ? kthread+0x0/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.838110] [] ? child_rip+0x0/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.843777] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.865194] Call Trace: Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.868147] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.875543] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.886277] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.893875] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.901337] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.908924] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.916570] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.923959] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.931037] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.937898] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.945742] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.952905] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.961237] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.967642] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.974733] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.982659] [] ? strlcpy+0x4a/0x60 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.988331] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478976.995177] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.002492] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.009686] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.016943] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.024583] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.032692] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.040033] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.046983] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.054878] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.062012] [] ? thread_return+0x4e/0x7d0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.068268] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.075118] [] ? kthread+0x9e/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.080708] [] ? child_rip+0xa/0x20 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.086375] [] ? kthread+0x0/0xc0 Aug 25 19:05:15 feral28.ccs.ornl.gov kernel: [1478977.091949] [] ? child_rip+0x0/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.745521] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748521] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748523] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748565] CPU 5 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748566] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748602] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748603] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748606] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748610] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748612] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748614] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748615] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748617] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748618] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748620] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748622] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748623] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748625] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748627] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748629] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748630] Stack: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748631] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748633] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748636] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748638] Call Trace: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748682] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748728] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748778] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748825] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748872] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748887] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748898] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748910] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748921] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748936] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748952] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748981] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.748996] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749011] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749061] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749065] [] ? strlcpy+0x4a/0x60 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749082] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749096] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749111] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749126] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749170] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749210] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749264] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749319] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749372] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749424] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749428] [] ? thread_return+0x4e/0x7d0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749479] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749483] [] ? kthread+0x9e/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749486] [] ? child_rip+0xa/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749488] [] ? kthread+0x0/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749490] [] ? child_rip+0x0/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749492] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749507] Call Trace: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749549] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749596] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749645] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749692] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749738] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749754] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749764] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749776] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749787] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749802] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749818] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749847] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749862] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749877] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749927] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749931] [] ? strlcpy+0x4a/0x60 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749947] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749962] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749977] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.749991] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750035] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750076] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750130] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750184] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750237] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750289] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750293] [] ? thread_return+0x4e/0x7d0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750345] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750348] [] ? kthread+0x9e/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750351] [] ? child_rip+0xa/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750354] [] ? kthread+0x0/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.750356] [] ? child_rip+0x0/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752521] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752523] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752566] CPU 6 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752566] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752603] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752604] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752607] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752612] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752613] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752615] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752617] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752618] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752620] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752622] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752624] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752625] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752629] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752631] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752632] Stack: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752633] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752635] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752638] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752640] Call Trace: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752675] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752722] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752772] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752818] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752865] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752881] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752892] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752904] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752916] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752966] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752977] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.752993] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753022] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753038] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753067] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753082] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753097] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753147] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753151] [] ? strlcpy+0x4a/0x60 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753167] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753182] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753197] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753212] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753256] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753297] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753351] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753405] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753458] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753510] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753514] [] ? thread_return+0x4e/0x7d0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753566] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753569] [] ? kthread+0x9e/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753572] [] ? child_rip+0xa/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753575] [] ? kthread+0x0/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753577] [] ? child_rip+0x0/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753578] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753593] Call Trace: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753636] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753682] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753732] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753779] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753825] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753841] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753851] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753863] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753875] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753925] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753936] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753951] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753980] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.753997] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754025] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754040] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754055] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754105] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754109] [] ? strlcpy+0x4a/0x60 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754125] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754140] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754155] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754170] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754213] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754254] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754308] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754361] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754415] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754467] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754470] [] ? thread_return+0x4e/0x7d0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754522] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754525] [] ? kthread+0x9e/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754528] [] ? child_rip+0xa/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754531] [] ? kthread+0x0/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.754533] [] ? child_rip+0x0/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755520] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755522] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755564] CPU 7 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755565] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755601] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755603] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755605] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755610] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755611] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755613] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755615] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755616] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755618] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755620] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755622] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755624] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755626] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755627] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755629] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755630] Stack: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755631] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755633] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755635] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755638] Call Trace: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755672] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755718] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755768] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755815] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755861] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755877] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755888] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755900] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755911] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755926] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755942] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755971] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.755986] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756001] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756051] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756055] [] ? strlcpy+0x4a/0x60 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756071] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756086] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756101] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756115] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756159] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756200] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756254] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756307] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756361] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756413] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756416] [] ? thread_return+0x4e/0x7d0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756468] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756471] [] ? kthread+0x9e/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756474] [] ? child_rip+0xa/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756477] [] ? kthread+0x0/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756479] [] ? child_rip+0x0/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756481] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756496] Call Trace: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756538] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756585] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756634] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756681] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756727] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756743] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756753] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756765] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756776] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756791] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756807] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756836] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756850] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756865] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756915] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756919] [] ? strlcpy+0x4a/0x60 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756935] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756950] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756965] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.756980] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757024] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757064] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757118] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757172] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757225] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757277] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757280] [] ? thread_return+0x4e/0x7d0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757332] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757335] [] ? kthread+0x9e/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757338] [] ? child_rip+0xa/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757341] [] ? kthread+0x0/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479057.757343] [] ? child_rip+0x0/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.274821] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.372106] CPU 4 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.374204] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.471829] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.473702] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.487135] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.495785] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.501653] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.509635] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.517591] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.525539] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.533495] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.541374] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.550309] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.556654] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.564627] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.572488] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.580410] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.589969] Stack: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.592518] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.600396] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.608950] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.617807] Call Trace: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.620762] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.628148] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.636221] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.646573] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.654133] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.661660] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.669308] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.676667] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.683747] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.690666] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.698573] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.705814] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.714163] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.720560] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.727570] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.735541] [] ? strlcpy+0x4a/0x60 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.741137] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.747956] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.755315] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.762508] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.769825] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.777353] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.785376] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.792855] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.799806] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.807691] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.814784] [] ? thread_return+0x4e/0x7d0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.821133] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.827967] [] ? kthread+0x9e/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.833609] [] ? child_rip+0xa/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.839303] [] ? kthread+0x0/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.844841] [] ? child_rip+0x0/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.850507] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.871975] Call Trace: Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.874921] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.882281] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.890337] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.898021] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.905571] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.913106] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.920772] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.928217] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.935305] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.942217] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.950062] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.957278] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.965635] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.972032] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.979131] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.987067] [] ? strlcpy+0x4a/0x60 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.992695] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479060.999507] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.006839] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.014058] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.021445] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.028973] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.037021] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.044370] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.051330] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.059198] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.066300] [] ? thread_return+0x4e/0x7d0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.072614] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.079517] [] ? kthread+0x9e/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.085107] [] ? child_rip+0xa/0x20 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.090809] [] ? kthread+0x0/0xc0 Aug 25 19:06:39 feral28.ccs.ornl.gov kernel: [1479061.096287] [] ? child_rip+0x0/0x20 Aug 25 19:06:43 feral28.ccs.ornl.gov kernel: [1479065.757312] Lustre: sultan-MDT000b: Client 1f0727a3-d49c-044b-6def-11f5d586ab1d (at 26@gni1) reconnecting Aug 25 19:06:43 feral28.ccs.ornl.gov kernel: [1479065.767718] Lustre: Skipped 4 previous similar messages Aug 25 19:06:43 feral28.ccs.ornl.gov kernel: [1479065.773581] Lustre: sultan-MDT000b: Connection restored to c60f26de-f3d6-6cfc-b87c-120a7b1f0eed (at 26@gni1) Aug 25 19:06:43 feral28.ccs.ornl.gov kernel: [1479065.784266] Lustre: Skipped 7 previous similar messages Aug 25 19:07:47 feral28.ccs.ornl.gov kernel: [1479129.265405] Lustre: sultan-MDT000b: haven't heard from client 2d302f5b-b294-591e-d666-ff70376d37f3 (at 10@gni1) in 340 seconds. I think it's dead, and I am evicting it. exp ffff880bb8cf8000, cur 1472166467 expire 1472166245 last 1472166127 Aug 25 19:07:47 feral28.ccs.ornl.gov kernel: [1479129.288042] Lustre: Skipped 1 previous similar message Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.752137] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755137] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755138] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755180] CPU 5 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755181] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755217] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755218] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755221] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755225] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755227] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755229] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755230] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755232] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755233] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755235] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755237] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755239] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755242] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755244] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755245] Stack: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755246] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755248] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755251] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755253] Call Trace: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755297] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755343] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755393] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755440] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755487] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755502] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755513] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755525] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755535] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755551] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755567] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755596] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755611] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755626] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755676] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755680] [] ? strlcpy+0x4a/0x60 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755696] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755711] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755726] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755741] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755785] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755825] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755880] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755934] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.755987] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756039] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756042] [] ? thread_return+0x4e/0x7d0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756094] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756098] [] ? kthread+0x9e/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756101] [] ? child_rip+0xa/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756103] [] ? kthread+0x0/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756106] [] ? child_rip+0x0/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756107] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756122] Call Trace: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756165] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756211] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756261] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756307] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756354] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756370] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756380] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756392] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756403] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756418] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756434] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756463] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756477] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756492] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756542] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756546] [] ? strlcpy+0x4a/0x60 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756562] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756577] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756592] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756607] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756651] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756691] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756745] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756799] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756853] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756904] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756908] [] ? thread_return+0x4e/0x7d0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756960] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756963] [] ? kthread+0x9e/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756966] [] ? child_rip+0xa/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756969] [] ? kthread+0x0/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.756971] [] ? child_rip+0x0/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759137] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759139] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759182] CPU 6 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759183] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759220] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759222] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759225] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759229] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759231] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759232] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759234] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759235] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759237] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759239] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759241] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759243] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759244] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759246] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759248] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759250] Stack: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759251] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759253] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759255] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759257] Call Trace: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759292] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759339] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759389] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759435] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759482] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759498] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759509] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759520] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759533] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759583] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759594] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759609] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759640] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759657] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759686] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759700] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759716] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759766] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759770] [] ? strlcpy+0x4a/0x60 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759786] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759801] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759816] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759831] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759874] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759915] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.759969] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760023] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760077] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760129] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760132] [] ? thread_return+0x4e/0x7d0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760184] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760188] [] ? kthread+0x9e/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760191] [] ? child_rip+0xa/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760193] [] ? kthread+0x0/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760196] [] ? child_rip+0x0/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760197] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760212] Call Trace: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760255] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760301] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760351] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760398] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760444] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760460] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760471] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760482] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760495] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760545] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760555] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760570] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760599] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760616] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760644] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760659] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760674] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760724] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760728] [] ? strlcpy+0x4a/0x60 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760744] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760759] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760774] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760789] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760832] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760873] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760926] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.760980] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.761034] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.761086] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.761089] [] ? thread_return+0x4e/0x7d0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.761141] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.761145] [] ? kthread+0x9e/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.761147] [] ? child_rip+0xa/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.761150] [] ? kthread+0x0/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.761152] [] ? child_rip+0x0/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762136] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762138] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762178] CPU 7 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762179] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762214] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762216] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762219] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762224] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762225] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762227] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762229] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762230] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762232] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762234] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762236] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762238] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762239] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762241] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762243] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762244] Stack: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762245] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762247] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762249] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762251] Call Trace: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762285] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762331] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762381] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762428] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762475] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762490] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762501] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762513] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762523] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762538] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762554] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762583] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762598] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762613] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762663] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762667] [] ? strlcpy+0x4a/0x60 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762683] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762698] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762713] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762728] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762772] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762812] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762866] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762920] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.762973] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763025] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763029] [] ? thread_return+0x4e/0x7d0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763081] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763084] [] ? kthread+0x9e/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763087] [] ? child_rip+0xa/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763090] [] ? kthread+0x0/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763092] [] ? child_rip+0x0/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763093] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763108] Call Trace: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763150] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763197] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763246] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763293] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763340] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763355] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763366] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763378] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763388] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763403] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763419] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763448] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763462] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763477] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763528] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763532] [] ? strlcpy+0x4a/0x60 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763548] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763562] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763577] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763592] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763636] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763676] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763730] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763784] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763838] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763890] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763893] [] ? thread_return+0x4e/0x7d0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763945] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763948] [] ? kthread+0x9e/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763951] [] ? child_rip+0xa/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763954] [] ? kthread+0x0/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479141.763956] [] ? child_rip+0x0/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.281846] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.378959] CPU 4 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.381099] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.481428] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.483292] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.496847] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.505462] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.511391] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.519312] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.527268] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.535277] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.543198] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.551164] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.560081] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.566409] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.574383] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.582295] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.590268] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.599724] Stack: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.602150] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.610071] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.618695] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.627484] Call Trace: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.630412] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.637763] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.645880] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.653599] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.661114] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.668632] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.676289] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.683648] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.690726] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.697597] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.705493] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.712708] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.720988] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.727359] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.734345] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.742264] [] ? strlcpy+0x4a/0x60 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.747884] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.754695] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.762017] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.769168] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.776416] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.783980] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.792037] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.799506] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.806457] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.814308] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.821349] [] ? thread_return+0x4e/0x7d0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.827673] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.834515] [] ? kthread+0x9e/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.840122] [] ? child_rip+0xa/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.845816] [] ? kthread+0x0/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.851318] [] ? child_rip+0x0/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.857053] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.878627] Call Trace: Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.881521] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.888890] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.896946] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.904586] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.911989] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.919496] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.927110] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.934436] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.941437] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.948272] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.956071] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.963210] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.971481] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.977789] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.984786] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.992695] [] ? strlcpy+0x4a/0x60 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479144.998360] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.005110] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.012511] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.019696] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.027013] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.034594] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.042650] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.050095] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.057054] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.064958] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.072007] [] ? thread_return+0x4e/0x7d0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.078321] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.085155] [] ? kthread+0x9e/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.090769] [] ? child_rip+0xa/0x20 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.096465] [] ? kthread+0x0/0xc0 Aug 25 19:08:03 feral28.ccs.ornl.gov kernel: [1479145.101925] [] ? child_rip+0x0/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.758753] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761753] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761755] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761797] CPU 5 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761798] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761834] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761835] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761838] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761843] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761844] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761846] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761847] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761849] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761851] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761853] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761854] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761856] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761858] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761859] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761861] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761863] Stack: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761863] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761866] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761868] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761870] Call Trace: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761914] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.761961] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762010] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762057] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762104] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762120] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762130] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762143] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762153] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762168] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762184] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762213] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762228] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762243] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762294] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762298] [] ? strlcpy+0x4a/0x60 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762314] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762329] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762343] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762358] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762402] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762442] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762497] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762550] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762604] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762655] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762659] [] ? thread_return+0x4e/0x7d0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762710] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762714] [] ? kthread+0x9e/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762717] [] ? child_rip+0xa/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762720] [] ? kthread+0x0/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762722] [] ? child_rip+0x0/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762723] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762738] Call Trace: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762781] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762828] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762877] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762924] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762970] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762986] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.762997] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763009] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763019] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763034] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763050] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763079] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763094] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763109] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763159] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763163] [] ? strlcpy+0x4a/0x60 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763179] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763193] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763208] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763223] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763267] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763307] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763361] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763415] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763468] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763520] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763524] [] ? thread_return+0x4e/0x7d0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763576] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763579] [] ? kthread+0x9e/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763582] [] ? child_rip+0xa/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763584] [] ? kthread+0x0/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.763587] [] ? child_rip+0x0/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765753] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765755] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765797] CPU 6 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765798] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765835] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765837] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765840] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765844] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765846] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765848] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765849] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765851] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765852] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765854] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765856] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765858] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765860] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765864] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765865] Stack: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765866] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765868] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765870] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765873] Call Trace: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765907] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.765954] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766004] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766051] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766097] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766113] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766124] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766135] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766148] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766198] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766209] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766224] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766253] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766270] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766298] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766313] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766328] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766378] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766382] [] ? strlcpy+0x4a/0x60 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766399] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766413] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766428] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766443] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766487] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766527] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766581] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766635] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766689] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766741] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766744] [] ? thread_return+0x4e/0x7d0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766796] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766799] [] ? kthread+0x9e/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766802] [] ? child_rip+0xa/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766805] [] ? kthread+0x0/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766807] [] ? child_rip+0x0/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766808] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766824] Call Trace: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766866] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766913] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.766962] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767009] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767055] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767071] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767082] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767093] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767105] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767155] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767166] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767181] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767210] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767227] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767255] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767270] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767285] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767335] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767339] [] ? strlcpy+0x4a/0x60 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767355] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767369] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767384] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767399] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767443] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767484] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767538] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767591] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767645] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767697] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767700] [] ? thread_return+0x4e/0x7d0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767753] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767756] [] ? kthread+0x9e/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767759] [] ? child_rip+0xa/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767762] [] ? kthread+0x0/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.767764] [] ? child_rip+0x0/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768752] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768754] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768793] CPU 7 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768794] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768830] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768831] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768834] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768838] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768840] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768842] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768843] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768845] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768847] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768849] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768851] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768852] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768854] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768856] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768858] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768859] Stack: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768860] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768862] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768864] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768866] Call Trace: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768900] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768946] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.768996] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769042] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769089] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769105] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769115] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769127] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769138] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769153] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769169] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769198] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769213] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769228] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769278] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769281] [] ? strlcpy+0x4a/0x60 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769298] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769312] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769327] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769342] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769386] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769426] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769480] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769534] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769587] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769639] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769642] [] ? thread_return+0x4e/0x7d0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769694] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769697] [] ? kthread+0x9e/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769700] [] ? child_rip+0xa/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769703] [] ? kthread+0x0/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769705] [] ? child_rip+0x0/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769707] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769722] Call Trace: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769764] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769811] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769860] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769907] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769953] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769969] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769979] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.769991] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770002] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770017] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770033] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770062] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770076] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770091] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770141] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770145] [] ? strlcpy+0x4a/0x60 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770161] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770176] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770191] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770206] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770249] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770290] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770344] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770398] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770451] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770503] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770507] [] ? thread_return+0x4e/0x7d0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770558] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770562] [] ? kthread+0x9e/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770564] [] ? child_rip+0xa/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770567] [] ? kthread+0x0/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479225.770569] [] ? child_rip+0x0/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.292328] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.389753] CPU 4 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.391858] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.489363] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.491254] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.504669] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.513362] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.519239] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.527185] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.535091] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.542986] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.550925] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.558795] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.567728] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.574099] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.582039] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.590013] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.597891] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.607399] Stack: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.609816] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.617790] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.626275] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.635072] Call Trace: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.638001] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.645439] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.653510] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.661170] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.668719] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.676307] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.683955] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.691305] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.698437] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.705375] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.713298] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.720469] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.728766] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.735120] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.742185] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.750120] [] ? strlcpy+0x4a/0x60 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.755774] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.762621] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.770004] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.777215] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.784524] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.792139] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.800161] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.807572] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.814530] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.822424] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.829501] [] ? thread_return+0x4e/0x7d0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.835771] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.842726] [] ? kthread+0x9e/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.848298] [] ? child_rip+0xa/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.853975] [] ? kthread+0x0/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.859444] [] ? child_rip+0x0/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.865118] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.886612] Call Trace: Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.889515] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.896929] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.904958] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.912599] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.920123] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.927632] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.935236] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.942631] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.949805] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.956795] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.964632] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.971802] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.980144] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.986480] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479228.993483] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.001445] [] ? strlcpy+0x4a/0x60 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.007118] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.013902] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.021347] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.028548] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.035815] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.043447] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.051435] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.058870] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.065855] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.073697] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.080748] [] ? thread_return+0x4e/0x7d0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.087062] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.093895] [] ? kthread+0x9e/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.099503] [] ? child_rip+0xa/0x20 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.105222] [] ? kthread+0x0/0xc0 Aug 25 19:09:27 feral28.ccs.ornl.gov kernel: [1479229.110769] [] ? child_rip+0x0/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.765369] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768369] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768371] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768412] CPU 5 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768413] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768449] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768451] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768454] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768458] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768460] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768461] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768463] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768464] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768466] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768468] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768470] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768471] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768473] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768475] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768476] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768478] Stack: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768479] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768481] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768483] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768486] Call Trace: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768529] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768576] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768625] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768672] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768718] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768734] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768745] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768757] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768767] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768783] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768799] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768828] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768843] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768858] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768908] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768912] [] ? strlcpy+0x4a/0x60 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768928] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768943] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768958] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.768972] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769016] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769056] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769110] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769164] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769218] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769270] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769273] [] ? thread_return+0x4e/0x7d0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769325] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769328] [] ? kthread+0x9e/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769331] [] ? child_rip+0xa/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769334] [] ? kthread+0x0/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769336] [] ? child_rip+0x0/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769337] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769352] Call Trace: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769395] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769441] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769491] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769537] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769584] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769600] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769610] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769622] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769632] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769648] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769664] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769692] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769707] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769722] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769772] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769776] [] ? strlcpy+0x4a/0x60 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769792] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769806] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769821] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769836] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769880] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769921] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.769975] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.770029] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.770082] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.770134] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.770137] [] ? thread_return+0x4e/0x7d0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.770189] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.770192] [] ? kthread+0x9e/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.770195] [] ? child_rip+0xa/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.770198] [] ? kthread+0x0/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.770200] [] ? child_rip+0x0/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772369] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772371] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772413] CPU 6 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772414] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772451] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772453] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772456] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772461] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772463] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772464] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772466] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772468] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772469] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772471] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772473] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772475] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772477] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772478] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772480] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772482] Stack: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772483] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772485] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772487] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772489] Call Trace: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772524] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772571] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772621] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772667] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772714] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772730] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772741] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772752] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772764] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772815] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772825] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772841] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772870] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772886] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772915] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772930] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772945] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772995] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.772999] [] ? strlcpy+0x4a/0x60 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773015] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773030] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773045] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773060] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773104] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773144] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773198] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773252] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773306] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773358] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773361] [] ? thread_return+0x4e/0x7d0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773413] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773416] [] ? kthread+0x9e/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773419] [] ? child_rip+0xa/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773422] [] ? kthread+0x0/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773424] [] ? child_rip+0x0/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773425] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773440] Call Trace: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773483] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773529] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773579] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773626] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773672] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773688] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773698] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773710] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773722] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773772] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773783] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773798] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773829] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773846] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773875] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773889] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773904] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773955] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773959] [] ? strlcpy+0x4a/0x60 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773975] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.773990] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774005] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774019] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774063] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774104] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774158] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774211] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774265] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774317] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774320] [] ? thread_return+0x4e/0x7d0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774372] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774376] [] ? kthread+0x9e/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774378] [] ? child_rip+0xa/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774381] [] ? kthread+0x0/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.774383] [] ? child_rip+0x0/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775369] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775370] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775409] CPU 7 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775410] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775446] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775447] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775450] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775454] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775456] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775457] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775459] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775461] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775462] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775464] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775466] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775468] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775470] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775472] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775474] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775475] Stack: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775476] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775478] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775480] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775482] Call Trace: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775516] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775563] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775612] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775659] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775705] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775721] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775732] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775744] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775754] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775769] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775786] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775815] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775829] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775844] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775895] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775899] [] ? strlcpy+0x4a/0x60 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775915] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775930] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775944] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.775959] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776003] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776043] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776097] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776151] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776204] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776256] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776260] [] ? thread_return+0x4e/0x7d0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776312] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776315] [] ? kthread+0x9e/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776318] [] ? child_rip+0xa/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776321] [] ? kthread+0x0/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776323] [] ? child_rip+0x0/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776324] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776339] Call Trace: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776382] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776428] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776478] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776524] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776571] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776587] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776597] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776609] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776620] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776635] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776651] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776680] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776694] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776709] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776759] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776763] [] ? strlcpy+0x4a/0x60 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776779] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776794] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776809] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776823] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776867] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776907] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.776961] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.777015] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.777069] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.777121] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.777124] [] ? thread_return+0x4e/0x7d0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.777176] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.777180] [] ? kthread+0x9e/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.777182] [] ? child_rip+0xa/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.777185] [] ? kthread+0x0/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479309.777187] [] ? child_rip+0x0/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.299196] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.396291] CPU 4 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.398416] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.495954] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.497819] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.511312] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.519944] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.525821] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.533742] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.541664] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.549576] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.557507] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.565411] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.574322] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.580726] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.588629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.596525] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.604402] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.613816] Stack: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.616225] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.624122] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.632684] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.641481] Call Trace: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.644349] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.651786] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.659816] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.667474] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.675016] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.682543] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.690216] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.697593] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.704691] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.711602] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.719420] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.726651] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.735027] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.741391] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.748438] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.756347] [] ? strlcpy+0x4a/0x60 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.762000] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.768847] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.776214] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.783469] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.790778] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.798383] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.806492] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.813885] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.820878] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.828764] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.835875] [] ? thread_return+0x4e/0x7d0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.842179] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.849015] [] ? kthread+0x9e/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.854613] [] ? child_rip+0xa/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.860316] [] ? kthread+0x0/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.865818] [] ? child_rip+0x0/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.871492] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.892962] Call Trace: Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.895872] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.903268] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.911385] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.919087] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.926522] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.934136] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.941739] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.949108] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.956206] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.963170] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.973623] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.980855] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.989109] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479312.995454] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.002432] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.010463] [] ? strlcpy+0x4a/0x60 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.016125] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.022937] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.030313] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.037480] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.044780] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.052395] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.060461] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.067853] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.074787] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.082645] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.089695] [] ? thread_return+0x4e/0x7d0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.095993] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.102887] [] ? kthread+0x9e/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.108501] [] ? child_rip+0xa/0x20 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.114197] [] ? kthread+0x0/0xc0 Aug 25 19:10:51 feral28.ccs.ornl.gov kernel: [1479313.119733] [] ? child_rip+0x0/0x20 Aug 25 19:11:05 feral28.ccs.ornl.gov kernel: [1479327.232260] Lustre: sultan-MDT000b: haven't heard from client 5d9f30ae-edef-9a3d-c88a-aa425ee6b9cf (at 15@gni1) in 229 seconds. I think it's dead, and I am evicting it. exp ffff880bbc47f800, cur 1472166665 expire 1472166443 last 1472166436 Aug 25 19:11:05 feral28.ccs.ornl.gov kernel: [1479327.255060] Lustre: Skipped 1 previous similar message Aug 25 19:11:28 feral28.ccs.ornl.gov kernel: [1479350.782800] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 19:11:28 feral28.ccs.ornl.gov kernel: [1479350.793955] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 30 previous similar messages Aug 25 19:11:28 feral28.ccs.ornl.gov kernel: [1479350.805085] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.202.61@o2ib1 (224): c: 63, oc: 0, rc: 63 Aug 25 19:11:28 feral28.ccs.ornl.gov kernel: [1479350.818439] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 30 previous similar messages Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.771986] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.774985] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.774987] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775029] CPU 5 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775030] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775065] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775067] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775070] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775074] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775076] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775078] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775079] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775081] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775082] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775084] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775086] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775088] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775091] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775093] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775094] Stack: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775095] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775097] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775100] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775102] Call Trace: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775146] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775192] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775242] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775289] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775336] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775351] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775362] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775374] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775385] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775400] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775416] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775445] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775460] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775475] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775526] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775530] [] ? strlcpy+0x4a/0x60 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775546] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775561] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775576] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775590] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775634] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775674] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775728] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775782] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775835] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775887] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775891] [] ? thread_return+0x4e/0x7d0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775943] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775946] [] ? kthread+0x9e/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775949] [] ? child_rip+0xa/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775951] [] ? kthread+0x0/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775954] [] ? child_rip+0x0/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775955] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.775970] Call Trace: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776013] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776059] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776109] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776155] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776202] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776217] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776228] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776240] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776250] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776265] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776281] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776310] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776324] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776339] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776389] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776393] [] ? strlcpy+0x4a/0x60 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776409] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776424] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776439] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776454] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776498] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776546] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776600] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776654] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776707] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776759] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776762] [] ? thread_return+0x4e/0x7d0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776814] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776818] [] ? kthread+0x9e/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776820] [] ? child_rip+0xa/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776823] [] ? kthread+0x0/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.776825] [] ? child_rip+0x0/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.778986] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.778987] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779030] CPU 6 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779031] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779067] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779069] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779072] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779077] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779078] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779080] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779082] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779083] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779085] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779087] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779089] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779090] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779094] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779096] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779097] Stack: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779098] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779100] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779102] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779105] Call Trace: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779140] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779186] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779236] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779283] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779329] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779345] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779356] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779368] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779380] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779430] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779441] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779456] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779488] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779504] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779533] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779548] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779563] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779613] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779617] [] ? strlcpy+0x4a/0x60 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779634] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779648] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779663] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779678] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779722] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779762] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779817] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779870] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779924] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779976] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.779980] [] ? thread_return+0x4e/0x7d0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780031] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780035] [] ? kthread+0x9e/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780038] [] ? child_rip+0xa/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780040] [] ? kthread+0x0/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780043] [] ? child_rip+0x0/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780044] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780059] Call Trace: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780102] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780148] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780198] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780245] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780291] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780307] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780317] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780329] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780341] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780391] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780402] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780417] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780446] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780462] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780491] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780506] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780521] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780571] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780575] [] ? strlcpy+0x4a/0x60 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780591] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780606] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780621] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780636] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780679] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780720] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780774] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780828] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780881] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780933] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780937] [] ? thread_return+0x4e/0x7d0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780988] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780992] [] ? kthread+0x9e/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780995] [] ? child_rip+0xa/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.780998] [] ? kthread+0x0/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.781000] [] ? child_rip+0x0/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.781984] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.781986] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782025] CPU 7 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782026] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782061] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782063] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782066] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782070] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782072] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782074] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782075] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782077] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782079] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782081] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782083] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782084] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782086] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782088] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782090] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782091] Stack: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782092] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782094] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782096] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782098] Call Trace: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782132] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782178] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782228] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782275] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782321] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782337] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782348] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782360] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782370] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782385] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782401] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782430] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782445] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782460] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782510] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782514] [] ? strlcpy+0x4a/0x60 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782530] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782545] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782560] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782574] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782618] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782659] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782712] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782766] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782819] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782871] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782875] [] ? thread_return+0x4e/0x7d0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782926] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782930] [] ? kthread+0x9e/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782933] [] ? child_rip+0xa/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782936] [] ? kthread+0x0/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782938] [] ? child_rip+0x0/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782939] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782954] Call Trace: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.782997] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783043] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783093] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783139] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783186] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783201] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783212] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783224] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783235] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783249] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783266] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783294] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783309] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783324] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783374] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783378] [] ? strlcpy+0x4a/0x60 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783394] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783408] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783423] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783438] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783482] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783522] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783576] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783630] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783683] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783735] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783738] [] ? thread_return+0x4e/0x7d0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783790] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783793] [] ? kthread+0x9e/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783796] [] ? child_rip+0xa/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783799] [] ? kthread+0x0/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479393.783801] [] ? child_rip+0x0/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.303101] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.400455] CPU 4 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.402588] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.500110] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.501956] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.515379] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.524013] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.529907] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.537803] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.545698] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.553619] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.561533] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.569446] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.578416] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.584734] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.592639] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.600541] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.608491] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.617911] Stack: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.620321] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.628232] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.636727] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.645558] Call Trace: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.651147] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.658561] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.666599] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.674255] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.681763] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.689341] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.696946] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.704323] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.711420] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.718297] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.726150] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.733314] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.741567] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.747878] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.754933] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.762868] [] ? strlcpy+0x4a/0x60 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.768531] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.775378] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.782780] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.789929] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.797239] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.804828] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.812936] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.820329] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.827297] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.835156] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.842180] [] ? thread_return+0x4e/0x7d0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.848433] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.855379] [] ? kthread+0x9e/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.861005] [] ? child_rip+0xa/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.866724] [] ? kthread+0x0/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.872245] [] ? child_rip+0x0/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.877964] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.899432] Call Trace: Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.902334] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.909790] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.917819] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.925504] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.933044] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.940614] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.948314] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.955657] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.962726] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.969672] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.977545] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.984716] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.993013] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479396.999359] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.006458] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.014365] [] ? strlcpy+0x4a/0x60 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.019978] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.026832] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.034173] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.041367] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.048720] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.056317] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.064409] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.071862] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.078812] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.086655] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.093739] [] ? thread_return+0x4e/0x7d0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.100063] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.107009] [] ? kthread+0x9e/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.112623] [] ? child_rip+0xa/0x20 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.118353] [] ? kthread+0x0/0xc0 Aug 25 19:12:15 feral28.ccs.ornl.gov kernel: [1479397.123864] [] ? child_rip+0x0/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.778602] BUG: soft lockup - CPU#4 stuck for 68s! [mdt01_002:21487] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781602] BUG: soft lockup - CPU#5 stuck for 68s! [mdt01_007:17158] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781603] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781646] CPU 5 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781647] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781683] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781684] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781687] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781691] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781693] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781695] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781696] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781698] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781699] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781701] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781703] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781705] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781706] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781708] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781710] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781711] Stack: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781712] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781714] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781716] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781719] Call Trace: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781762] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781809] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781859] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781905] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781952] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781967] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781978] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.781990] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782001] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782016] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782032] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782061] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782076] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782091] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782141] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782145] [] ? strlcpy+0x4a/0x60 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782161] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782176] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782191] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782206] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782250] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782290] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782344] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782398] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782451] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782503] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782507] [] ? thread_return+0x4e/0x7d0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782558] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782562] [] ? kthread+0x9e/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782565] [] ? child_rip+0xa/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782568] [] ? kthread+0x0/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782570] [] ? child_rip+0x0/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782571] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782586] Call Trace: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782629] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782675] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782725] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782772] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782818] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782833] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782844] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782856] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782866] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782881] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782897] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782926] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782941] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.782956] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783006] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783010] [] ? strlcpy+0x4a/0x60 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783026] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783041] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783055] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783070] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783114] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783154] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783208] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783262] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783316] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783368] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783371] [] ? thread_return+0x4e/0x7d0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783423] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783426] [] ? kthread+0x9e/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783429] [] ? child_rip+0xa/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783432] [] ? kthread+0x0/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.783434] [] ? child_rip+0x0/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785602] BUG: soft lockup - CPU#6 stuck for 68s! [mdt01_000:21485] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785603] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785646] CPU 6 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785647] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785684] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785686] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785689] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785693] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785695] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785697] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785698] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785700] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785702] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785704] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785706] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785708] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785711] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785713] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785715] Stack: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785715] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785717] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785720] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785722] Call Trace: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785757] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785804] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785854] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785900] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785947] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785963] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785973] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785985] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.785997] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786048] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786058] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786073] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786105] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786121] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786150] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786164] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786179] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786230] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786234] [] ? strlcpy+0x4a/0x60 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786250] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786265] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786280] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786294] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786338] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786378] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786433] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786487] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786541] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786593] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786596] [] ? thread_return+0x4e/0x7d0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786648] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786651] [] ? kthread+0x9e/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786654] [] ? child_rip+0xa/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786657] [] ? kthread+0x0/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786659] [] ? child_rip+0x0/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786660] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786676] Call Trace: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786718] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786765] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786814] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786861] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786907] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786923] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786933] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786945] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.786957] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787007] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787017] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787033] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787064] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787080] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787109] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787123] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787138] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787188] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787192] [] ? strlcpy+0x4a/0x60 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787209] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787223] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787238] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787253] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787297] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787337] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787391] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787445] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787498] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787550] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787554] [] ? thread_return+0x4e/0x7d0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787606] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787609] [] ? kthread+0x9e/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787612] [] ? child_rip+0xa/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787615] [] ? kthread+0x0/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.787617] [] ? child_rip+0x0/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788601] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788602] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788642] CPU 7 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788643] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788678] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788680] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788683] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788687] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788688] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788690] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788692] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788693] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788695] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788697] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788699] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788701] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788704] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788706] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788708] Stack: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788709] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788711] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788713] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788715] Call Trace: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788749] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788796] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788845] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788892] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788939] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788954] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788965] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788977] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.788987] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789003] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789019] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789048] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789062] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789077] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789128] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789132] [] ? strlcpy+0x4a/0x60 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789148] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789163] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789177] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789192] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789236] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789276] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789330] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789384] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789438] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789490] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789493] [] ? thread_return+0x4e/0x7d0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789545] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789548] [] ? kthread+0x9e/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789551] [] ? child_rip+0xa/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789554] [] ? kthread+0x0/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789556] [] ? child_rip+0x0/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789557] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789573] Call Trace: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789615] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789662] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789711] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789758] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789805] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789820] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789831] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789842] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789853] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789868] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789884] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789913] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789928] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789943] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789992] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.789996] [] ? strlcpy+0x4a/0x60 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790013] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790027] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790042] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790057] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790101] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790141] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790195] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790249] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790303] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790354] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790357] [] ? thread_return+0x4e/0x7d0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790409] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790413] [] ? kthread+0x9e/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790415] [] ? child_rip+0xa/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790418] [] ? kthread+0x0/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479477.790420] [] ? child_rip+0x0/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.310229] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.407523] CPU 4 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.409666] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.509810] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.511668] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.525229] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.533890] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.539766] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.547721] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.555635] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.563565] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.571539] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.579470] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.588395] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.594738] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.602706] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.610668] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.618609] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.628125] Stack: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.630517] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.638429] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.646932] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.655754] Call Trace: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.658641] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.665967] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.673996] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.681628] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.689089] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.696592] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.704187] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.711469] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.718470] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.725348] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.733157] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.740286] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.748497] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.754833] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.761854] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.769729] [] ? strlcpy+0x4a/0x60 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.775357] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.782159] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.789518] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.796692] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.803969] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.811557] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.819640] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.827102] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.834035] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.841903] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.848979] [] ? thread_return+0x4e/0x7d0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.855285] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.862162] [] ? kthread+0x9e/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.867760] [] ? child_rip+0xa/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.873411] [] ? kthread+0x0/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.878871] [] ? child_rip+0x0/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.884572] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.906092] Call Trace: Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.909012] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.916355] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.924385] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.931982] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.939445] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.946928] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.954514] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.961823] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.968876] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.975719] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.983538] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.990701] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479480.998954] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.005213] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.012216] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.020126] [] ? strlcpy+0x4a/0x60 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.025711] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.032531] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.039931] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.047152] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.054444] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.062007] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.070149] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.077534] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.084441] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.092318] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.099420] [] ? thread_return+0x4e/0x7d0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.105735] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.112680] [] ? kthread+0x9e/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.118279] [] ? child_rip+0xa/0x20 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.123921] [] ? kthread+0x0/0xc0 Aug 25 19:13:39 feral28.ccs.ornl.gov kernel: [1479481.129405] [] ? child_rip+0x0/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.785218] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788218] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788219] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788262] CPU 5 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788262] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788299] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788301] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788304] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788308] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788310] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788311] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788313] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788314] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788316] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788318] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788320] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788321] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788323] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788325] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788327] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788328] Stack: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788329] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788331] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788333] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788336] Call Trace: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788379] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788426] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788476] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788523] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788569] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788585] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788596] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788608] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788618] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788634] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788650] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788679] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788693] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788709] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788759] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788763] [] ? strlcpy+0x4a/0x60 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788779] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788794] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788809] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788824] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788868] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788908] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.788962] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789016] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789070] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789122] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789125] [] ? thread_return+0x4e/0x7d0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789177] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789180] [] ? kthread+0x9e/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789183] [] ? child_rip+0xa/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789186] [] ? kthread+0x0/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789188] [] ? child_rip+0x0/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789189] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789205] Call Trace: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789247] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789294] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789343] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789390] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789436] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789452] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789462] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789474] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789485] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789500] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789516] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789545] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789559] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789574] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789624] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789628] [] ? strlcpy+0x4a/0x60 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789644] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789659] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789674] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789689] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789733] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789773] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789827] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789881] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789934] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789986] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.789990] [] ? thread_return+0x4e/0x7d0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.790042] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.790045] [] ? kthread+0x9e/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.790048] [] ? child_rip+0xa/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.790051] [] ? kthread+0x0/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.790053] [] ? child_rip+0x0/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792218] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792219] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792262] CPU 6 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792263] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792299] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792301] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792304] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792308] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792310] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792311] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792313] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792315] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792316] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792318] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792320] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792322] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792323] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792325] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792327] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792328] Stack: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792329] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792331] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792334] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792336] Call Trace: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792371] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792417] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792467] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792514] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792561] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792576] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792587] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792599] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792611] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792662] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792672] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792688] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792717] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792733] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792762] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792777] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792792] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792842] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792846] [] ? strlcpy+0x4a/0x60 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792862] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792877] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792892] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792907] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792951] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.792991] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793046] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793099] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793153] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793205] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793208] [] ? thread_return+0x4e/0x7d0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793260] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793264] [] ? kthread+0x9e/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793266] [] ? child_rip+0xa/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793269] [] ? kthread+0x0/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793271] [] ? child_rip+0x0/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793272] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793288] Call Trace: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793331] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793377] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793427] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793473] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793520] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793535] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793546] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793557] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793569] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793620] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793630] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793646] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793675] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793691] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793719] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793734] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793749] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793799] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793803] [] ? strlcpy+0x4a/0x60 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793819] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793834] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793849] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793864] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793908] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.793948] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794002] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794056] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794110] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794162] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794165] [] ? thread_return+0x4e/0x7d0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794217] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794220] [] ? kthread+0x9e/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794223] [] ? child_rip+0xa/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794226] [] ? kthread+0x0/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.794228] [] ? child_rip+0x0/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795217] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795218] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795258] CPU 7 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795259] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795295] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795297] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795300] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795304] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795306] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795307] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795309] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795310] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795312] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795314] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795316] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795317] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795321] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795323] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795324] Stack: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795325] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795327] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795330] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795332] Call Trace: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795366] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795413] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795462] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795509] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795556] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795571] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795582] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795594] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795605] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795620] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795636] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795665] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795680] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795695] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795745] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795749] [] ? strlcpy+0x4a/0x60 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795765] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795780] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795795] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795810] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795853] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795894] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.795948] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796001] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796055] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796106] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796110] [] ? thread_return+0x4e/0x7d0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796161] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796165] [] ? kthread+0x9e/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796168] [] ? child_rip+0xa/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796171] [] ? kthread+0x0/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796173] [] ? child_rip+0x0/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796174] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796189] Call Trace: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796232] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796278] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796328] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796374] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796421] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796436] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796447] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796459] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796470] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796485] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796501] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796529] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796544] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796559] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796609] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796613] [] ? strlcpy+0x4a/0x60 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796629] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796644] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796659] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796673] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796717] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796758] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796812] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796865] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796919] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796970] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.796974] [] ? thread_return+0x4e/0x7d0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.797026] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.797029] [] ? kthread+0x9e/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.797032] [] ? child_rip+0xa/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.797035] [] ? kthread+0x0/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479561.797037] [] ? child_rip+0x0/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.319334] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.416532] CPU 4 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.418631] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.516334] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.518207] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.531744] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.540402] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.546304] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.554243] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.562147] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.570095] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.578050] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.585956] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.594892] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.601201] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.609164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.617095] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.625036] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.634515] Stack: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.636944] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.644934] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.653454] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.662346] Call Trace: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.665300] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.672705] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.680752] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.688391] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.695838] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.703416] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.711010] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.718389] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.725467] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.732398] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.740225] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.747388] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.755721] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.762091] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.769112] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.776987] [] ? strlcpy+0x4a/0x60 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.782676] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.789470] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.796889] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.804065] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.811400] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.818963] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.827080] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.834533] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.841415] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.849291] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.856394] [] ? thread_return+0x4e/0x7d0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.862717] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.869594] [] ? kthread+0x9e/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.875253] [] ? child_rip+0xa/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.880997] [] ? kthread+0x0/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.886562] [] ? child_rip+0x0/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.892272] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.913715] Call Trace: Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.916634] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.924071] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.932120] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.939768] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.947284] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.954819] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.962485] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.969818] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.976958] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.983896] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.991732] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479564.998937] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.007286] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.013685] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.020714] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.028598] [] ? strlcpy+0x4a/0x60 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.034192] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.040985] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.048369] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.055537] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.062810] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.070417] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.078457] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.085876] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.092870] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.100790] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.107935] [] ? thread_return+0x4e/0x7d0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.114232] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.121152] [] ? kthread+0x9e/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.126802] [] ? child_rip+0xa/0x20 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.132514] [] ? kthread+0x0/0xc0 Aug 25 19:15:03 feral28.ccs.ornl.gov kernel: [1479565.137991] [] ? child_rip+0x0/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.791834] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794834] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794835] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794878] CPU 5 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794879] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794916] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794918] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794921] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794925] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794927] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794929] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794930] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794932] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794934] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794936] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794938] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794939] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794941] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794943] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794945] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794946] Stack: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794947] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794949] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794951] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794953] Call Trace: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.794997] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795043] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795093] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795140] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795186] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795202] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795213] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795225] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795235] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795250] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795266] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795295] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795310] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795325] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795375] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795379] [] ? strlcpy+0x4a/0x60 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795395] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795410] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795425] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795440] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795484] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795525] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795579] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795633] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795686] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795738] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795742] [] ? thread_return+0x4e/0x7d0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795794] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795797] [] ? kthread+0x9e/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795800] [] ? child_rip+0xa/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795803] [] ? kthread+0x0/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795805] [] ? child_rip+0x0/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795806] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795821] Call Trace: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795864] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795911] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.795960] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796007] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796053] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796069] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796080] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796092] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796102] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796117] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796133] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796162] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796177] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796192] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796242] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796246] [] ? strlcpy+0x4a/0x60 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796262] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796277] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796292] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796306] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796350] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796390] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796444] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796498] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796552] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796604] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796608] [] ? thread_return+0x4e/0x7d0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796659] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796663] [] ? kthread+0x9e/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796666] [] ? child_rip+0xa/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796668] [] ? kthread+0x0/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.796671] [] ? child_rip+0x0/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798834] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798836] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798878] CPU 6 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798879] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798916] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798917] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798920] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798925] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798926] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798928] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798929] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798931] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798933] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798935] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798937] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798939] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798940] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798942] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798944] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798946] Stack: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798947] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798949] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798951] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798953] Call Trace: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.798988] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799035] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799084] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799131] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799178] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799193] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799204] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799216] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799228] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799278] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799288] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799304] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799333] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799350] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799378] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799393] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799408] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799458] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799462] [] ? strlcpy+0x4a/0x60 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799478] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799493] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799508] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799523] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799567] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799607] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799662] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799716] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799769] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799821] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799825] [] ? thread_return+0x4e/0x7d0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799876] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799880] [] ? kthread+0x9e/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799883] [] ? child_rip+0xa/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799885] [] ? kthread+0x0/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799888] [] ? child_rip+0x0/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799889] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799904] Call Trace: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799947] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.799993] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800043] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800090] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800136] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800152] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800162] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800174] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800186] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800237] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800247] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800263] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800293] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800310] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800338] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800353] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800368] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800419] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800423] [] ? strlcpy+0x4a/0x60 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800439] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800454] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800468] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800483] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800527] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800568] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800622] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800676] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800729] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800781] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800784] [] ? thread_return+0x4e/0x7d0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800836] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800839] [] ? kthread+0x9e/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800842] [] ? child_rip+0xa/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800845] [] ? kthread+0x0/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.800847] [] ? child_rip+0x0/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801833] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801835] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801876] CPU 7 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801877] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801914] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801916] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801919] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801923] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801924] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801926] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801928] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801929] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801931] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801933] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801935] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801937] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801941] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801943] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801944] Stack: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801945] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801947] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801949] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801952] Call Trace: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.801986] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802033] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802082] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802129] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802176] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802191] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802202] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802214] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802224] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802240] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802256] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802285] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802299] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802314] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802365] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802369] [] ? strlcpy+0x4a/0x60 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802385] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802400] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802415] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802429] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802473] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802514] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802568] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802621] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802675] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802727] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802730] [] ? thread_return+0x4e/0x7d0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802782] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802785] [] ? kthread+0x9e/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802788] [] ? child_rip+0xa/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802791] [] ? kthread+0x0/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802793] [] ? child_rip+0x0/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802794] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802809] Call Trace: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802852] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802898] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802948] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.802994] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803041] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803057] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803067] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803079] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803090] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803105] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803121] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803150] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803165] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803180] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803230] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803234] [] ? strlcpy+0x4a/0x60 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803250] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803265] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803279] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803294] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803338] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803378] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803432] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803486] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803539] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803591] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803594] [] ? thread_return+0x4e/0x7d0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803646] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803649] [] ? kthread+0x9e/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803652] [] ? child_rip+0xa/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803655] [] ? kthread+0x0/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479645.803657] [] ? child_rip+0x0/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.325102] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.422700] CPU 4 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.424832] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.522344] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.524226] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.537754] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.546404] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.552246] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.560159] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.568072] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.576045] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.584036] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.591967] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.600971] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.607281] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.615212] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.623090] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.631115] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.640786] Stack: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.643248] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.651179] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.659690] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.668573] Call Trace: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.671474] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.678941] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.687013] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.694706] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.702167] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.709704] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.717317] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.724712] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.731861] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.738746] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.746609] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.753798] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.762026] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.768352] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.775399] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.783360] [] ? strlcpy+0x4a/0x60 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.788981] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.795775] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.803142] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.810300] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.817602] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.825233] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.833308] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.840682] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.847660] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.855588] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.862671] [] ? thread_return+0x4e/0x7d0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.868988] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.875899] [] ? kthread+0x9e/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.881566] [] ? child_rip+0xa/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.887312] [] ? kthread+0x0/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.892824] [] ? child_rip+0x0/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.898569] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.920019] Call Trace: Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.922922] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.930335] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.938424] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.946084] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.953563] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.961124] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.968790] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.976184] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.983298] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.990193] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479648.997983] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.005192] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.013505] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.019912] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.027002] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.037631] [] ? strlcpy+0x4a/0x60 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.043254] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.050003] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.057292] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.064512] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.071839] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.079392] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.087441] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.094868] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.101802] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.109712] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.116762] [] ? thread_return+0x4e/0x7d0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.123138] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.130067] [] ? kthread+0x9e/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.135742] [] ? child_rip+0xa/0x20 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.141461] [] ? kthread+0x0/0xc0 Aug 25 19:16:27 feral28.ccs.ornl.gov kernel: [1479649.147009] [] ? child_rip+0x0/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.798450] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801450] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801452] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801494] CPU 5 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801495] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801532] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801534] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801537] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801541] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801543] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801545] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801546] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801548] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801550] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801552] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801554] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801555] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801557] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801559] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801561] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801562] Stack: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801563] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801565] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801567] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801569] Call Trace: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801613] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801660] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801710] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801756] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801803] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801818] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801829] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801841] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801852] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801867] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801883] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801912] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801927] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801942] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801992] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.801996] [] ? strlcpy+0x4a/0x60 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802013] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802027] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802042] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802057] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802101] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802141] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802195] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802249] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802303] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802355] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802359] [] ? thread_return+0x4e/0x7d0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802411] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802414] [] ? kthread+0x9e/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802417] [] ? child_rip+0xa/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802420] [] ? kthread+0x0/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802422] [] ? child_rip+0x0/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802423] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802438] Call Trace: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802480] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802527] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802577] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802623] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802670] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802685] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802696] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802708] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802718] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802733] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802749] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802778] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802793] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802808] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802858] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802862] [] ? strlcpy+0x4a/0x60 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802878] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802893] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802908] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802923] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.802966] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803007] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803060] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803114] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803168] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803220] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803223] [] ? thread_return+0x4e/0x7d0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803275] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803279] [] ? kthread+0x9e/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803282] [] ? child_rip+0xa/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803284] [] ? kthread+0x0/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.803286] [] ? child_rip+0x0/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805450] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805452] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805495] CPU 6 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805496] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805533] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805535] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805538] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805543] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805544] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805546] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805547] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805549] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805550] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805553] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805554] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805556] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805559] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805561] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805563] Stack: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805564] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805566] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805568] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805571] Call Trace: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805606] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805652] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805702] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805749] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805796] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805811] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805822] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805834] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805846] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805896] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805907] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805922] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805954] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805970] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.805999] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806014] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806029] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806079] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806083] [] ? strlcpy+0x4a/0x60 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806100] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806114] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806129] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806144] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806188] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806228] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806282] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806336] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806390] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806441] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806445] [] ? thread_return+0x4e/0x7d0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806497] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806500] [] ? kthread+0x9e/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806503] [] ? child_rip+0xa/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806506] [] ? kthread+0x0/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806508] [] ? child_rip+0x0/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806509] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806525] Call Trace: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806568] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806614] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806664] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806710] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806757] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806772] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806783] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806795] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806807] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806857] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806868] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806883] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806914] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806930] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806959] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806974] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.806989] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807039] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807043] [] ? strlcpy+0x4a/0x60 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807059] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807074] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807089] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807104] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807147] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807187] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807242] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807296] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807349] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807401] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807404] [] ? thread_return+0x4e/0x7d0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807456] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807460] [] ? kthread+0x9e/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807463] [] ? child_rip+0xa/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807466] [] ? kthread+0x0/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.807468] [] ? child_rip+0x0/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808450] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808451] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808490] CPU 7 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808491] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808527] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808528] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808531] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808535] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808537] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808538] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808540] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808542] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808543] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808545] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808547] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808548] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808552] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808554] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808555] Stack: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808556] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808558] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808560] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808563] Call Trace: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808597] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808643] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808693] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808739] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808786] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808802] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808813] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808824] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808835] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808850] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808866] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808895] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808910] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808925] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808975] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808979] [] ? strlcpy+0x4a/0x60 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.808996] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809010] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809025] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809040] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809084] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809124] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809178] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809232] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809285] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809337] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809341] [] ? thread_return+0x4e/0x7d0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809393] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809396] [] ? kthread+0x9e/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809399] [] ? child_rip+0xa/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809402] [] ? kthread+0x0/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809404] [] ? child_rip+0x0/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809405] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809420] Call Trace: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809463] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809509] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809559] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809606] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809652] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809668] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809678] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809690] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809701] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809716] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809732] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809760] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809775] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809790] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809840] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809844] [] ? strlcpy+0x4a/0x60 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809860] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809875] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809890] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809905] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809948] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.809989] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810043] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810097] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810150] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810202] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810205] [] ? thread_return+0x4e/0x7d0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810257] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810260] [] ? kthread+0x9e/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810263] [] ? child_rip+0xa/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810266] [] ? kthread+0x0/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.810268] [] ? child_rip+0x0/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.885673] Lustre: sultan-MDT000b: Client 1f0727a3-d49c-044b-6def-11f5d586ab1d (at 26@gni1) reconnecting Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.885678] Lustre: Skipped 9 previous similar messages Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.885714] Lustre: sultan-MDT000b: Connection restored to c60f26de-f3d6-6cfc-b87c-120a7b1f0eed (at 26@gni1) Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479729.885718] Lustre: Skipped 11 previous similar messages Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.363916] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.461218] CPU 4 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.463344] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.560917] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.562815] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.576326] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.584950] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.590878] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.598791] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.606730] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.614711] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.622704] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.630651] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.639605] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.646017] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.654007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.661948] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.669878] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.679262] Stack: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.681682] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.689663] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.698176] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.707050] Call Trace: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.709988] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.717349] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.725412] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.733052] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.740533] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.748111] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.755733] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.763101] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.770241] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.777180] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.785102] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.792301] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.803302] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.809673] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.816703] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.824656] [] ? strlcpy+0x4a/0x60 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.830301] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.837086] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.844462] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.851681] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.858956] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.866579] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.874679] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.882132] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.889031] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.896856] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.903949] [] ? thread_return+0x4e/0x7d0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.910314] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.917218] [] ? kthread+0x9e/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.922868] [] ? child_rip+0xa/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.928569] [] ? kthread+0x0/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.934074] [] ? child_rip+0x0/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.939715] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.961192] Call Trace: Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.964128] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.971542] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.979632] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.987316] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479732.994805] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.002366] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.010040] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.017434] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.024487] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.031391] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.039234] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.046372] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.054644] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.061006] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.068009] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.075928] [] ? strlcpy+0x4a/0x60 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.081574] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.088444] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.095847] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.103050] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.110342] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.117895] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.125978] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.133396] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.140330] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.148233] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.155343] [] ? thread_return+0x4e/0x7d0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.161630] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.168509] [] ? kthread+0x9e/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.174194] [] ? child_rip+0xa/0x20 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.179931] [] ? kthread+0x0/0xc0 Aug 25 19:17:51 feral28.ccs.ornl.gov kernel: [1479733.185521] [] ? child_rip+0x0/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.805067] BUG: soft lockup - CPU#4 stuck for 68s! [mdt01_002:21487] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808066] BUG: soft lockup - CPU#5 stuck for 68s! [mdt01_007:17158] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808068] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808110] CPU 5 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808111] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808147] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808149] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808152] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808156] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808158] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808160] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808161] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808163] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808165] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808167] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808169] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808170] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808172] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808174] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808176] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808177] Stack: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808178] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808180] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808182] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808184] Call Trace: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808228] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808274] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808324] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808371] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808417] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808433] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808444] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808456] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808466] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808481] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808498] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808527] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808541] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808556] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808607] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808610] [] ? strlcpy+0x4a/0x60 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808627] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808641] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808656] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808671] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808715] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808755] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808810] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808864] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808918] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808970] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.808973] [] ? thread_return+0x4e/0x7d0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809025] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809028] [] ? kthread+0x9e/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809031] [] ? child_rip+0xa/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809034] [] ? kthread+0x0/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809036] [] ? child_rip+0x0/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809037] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809053] Call Trace: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809095] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809142] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809191] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809238] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809284] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809300] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809310] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809322] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809333] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809348] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809364] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809393] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809407] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809422] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809472] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809476] [] ? strlcpy+0x4a/0x60 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809492] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809507] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809522] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809537] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809581] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809621] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809676] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809729] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809783] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809835] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809838] [] ? thread_return+0x4e/0x7d0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809890] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809894] [] ? kthread+0x9e/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809896] [] ? child_rip+0xa/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809899] [] ? kthread+0x0/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.809901] [] ? child_rip+0x0/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812066] BUG: soft lockup - CPU#6 stuck for 68s! [mdt01_000:21485] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812068] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812111] CPU 6 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812112] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812149] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812150] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812153] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812158] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812159] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812161] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812162] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812164] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812166] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812168] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812170] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812172] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812174] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812175] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812177] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812179] Stack: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812180] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812182] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812184] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812186] Call Trace: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812221] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812268] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812318] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812364] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812411] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812427] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812438] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812449] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812461] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812512] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812522] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812538] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812569] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812585] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812614] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812629] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812644] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812694] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812698] [] ? strlcpy+0x4a/0x60 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812715] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812729] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812744] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812759] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812803] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812843] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812898] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.812952] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813005] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813057] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813060] [] ? thread_return+0x4e/0x7d0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813112] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813115] [] ? kthread+0x9e/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813118] [] ? child_rip+0xa/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813121] [] ? kthread+0x0/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813123] [] ? child_rip+0x0/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813124] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813140] Call Trace: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813183] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813229] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813279] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813325] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813372] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813387] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813398] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813410] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813422] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813472] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813482] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813498] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813529] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813545] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813574] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813588] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813603] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813654] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813657] [] ? strlcpy+0x4a/0x60 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813674] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813688] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813703] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813718] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813762] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813802] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813856] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813910] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.813963] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.814015] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.814019] [] ? thread_return+0x4e/0x7d0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.814071] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.814074] [] ? kthread+0x9e/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.814077] [] ? child_rip+0xa/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.814080] [] ? kthread+0x0/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.814082] [] ? child_rip+0x0/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815066] BUG: soft lockup - CPU#7 stuck for 68s! [mdt01_001:21486] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815067] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815107] CPU 7 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815108] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815143] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815145] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815148] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815152] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815153] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815155] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815157] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815158] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815160] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815162] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815164] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815165] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815168] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815170] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815172] Stack: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815172] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815175] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815177] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815179] Call Trace: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815213] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815260] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815309] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815356] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815402] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815418] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815429] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815440] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815451] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815466] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815482] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815511] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815526] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815541] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815591] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815595] [] ? strlcpy+0x4a/0x60 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815612] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815626] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815641] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815656] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815700] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815740] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815794] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815848] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815902] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815954] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.815957] [] ? thread_return+0x4e/0x7d0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816009] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816012] [] ? kthread+0x9e/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816015] [] ? child_rip+0xa/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816018] [] ? kthread+0x0/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816020] [] ? child_rip+0x0/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816021] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816037] Call Trace: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816079] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816126] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816175] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816222] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816268] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816284] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816294] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816306] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816317] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816332] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816348] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816376] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816391] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816406] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816456] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816460] [] ? strlcpy+0x4a/0x60 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816476] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816491] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816506] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816521] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816565] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816605] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816659] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816712] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816766] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816818] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816821] [] ? thread_return+0x4e/0x7d0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816873] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816877] [] ? kthread+0x9e/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816879] [] ? child_rip+0xa/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816882] [] ? kthread+0x0/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479813.816884] [] ? child_rip+0x0/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.336636] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.433836] CPU 4 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.435908] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.533550] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.535423] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.548968] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.557662] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.563556] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.571434] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.582034] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.590033] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.597964] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.605929] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.614899] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.621208] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.629182] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.637087] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.645017] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.654472] Stack: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.656916] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.664890] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.673444] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.682293] Call Trace: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.685213] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.692695] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.700716] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.708417] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.715854] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.723397] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.731010] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.738405] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.745519] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.752447] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.760319] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.767543] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.775823] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.782211] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.789310] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.797289] [] ? strlcpy+0x4a/0x60 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.802883] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.809745] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.817148] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.824367] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.831642] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.839239] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.847313] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.854731] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.861700] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.869567] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.876713] [] ? thread_return+0x4e/0x7d0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.883071] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.889991] [] ? kthread+0x9e/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.895623] [] ? child_rip+0xa/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.901343] [] ? kthread+0x0/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.906829] [] ? child_rip+0x0/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.912540] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.933964] Call Trace: Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.936883] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.944245] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.952318] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.959968] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.967473] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.975052] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.982691] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.990058] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479816.997111] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.004025] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.011852] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.019085] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.027364] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.033761] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.040835] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.048770] [] ? strlcpy+0x4a/0x60 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.054390] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.061235] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.068585] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.075794] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.083071] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.090659] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.098750] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.106126] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.113110] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.120996] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.128098] [] ? thread_return+0x4e/0x7d0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.134439] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.141411] [] ? kthread+0x9e/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.146992] [] ? child_rip+0xa/0x20 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.152729] [] ? kthread+0x0/0xc0 Aug 25 19:19:15 feral28.ccs.ornl.gov kernel: [1479817.158233] [] ? child_rip+0x0/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.811683] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814683] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814684] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814726] CPU 5 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814727] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814763] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814765] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814768] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814772] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814774] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814775] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814777] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814779] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814780] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814782] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814785] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814786] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814788] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814790] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814791] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814793] Stack: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814794] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814796] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814798] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814800] Call Trace: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814843] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814890] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814940] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.814986] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815033] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815049] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815059] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815072] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815082] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815097] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815113] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815142] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815157] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815172] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815223] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815227] [] ? strlcpy+0x4a/0x60 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815243] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815258] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815272] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815287] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815331] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815371] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815426] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815480] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815533] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815585] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815589] [] ? thread_return+0x4e/0x7d0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815641] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815644] [] ? kthread+0x9e/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815647] [] ? child_rip+0xa/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815650] [] ? kthread+0x0/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815652] [] ? child_rip+0x0/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815653] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815668] Call Trace: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815711] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815757] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815807] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815854] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815900] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815916] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815926] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815938] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815949] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815964] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.815980] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816008] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816023] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816038] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816088] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816092] [] ? strlcpy+0x4a/0x60 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816108] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816123] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816138] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816153] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816197] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816237] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816291] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816344] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816398] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816450] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816453] [] ? thread_return+0x4e/0x7d0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816505] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816509] [] ? kthread+0x9e/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816511] [] ? child_rip+0xa/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816514] [] ? kthread+0x0/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.816516] [] ? child_rip+0x0/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818683] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818685] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818727] CPU 6 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818728] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818765] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818766] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818769] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818774] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818776] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818777] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818779] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818781] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818782] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818784] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818786] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818788] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818792] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818794] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818795] Stack: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818796] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818798] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818800] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818802] Call Trace: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818837] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818884] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818933] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.818980] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819027] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819042] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819053] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819064] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819077] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819127] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819137] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819153] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819182] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819198] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819227] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819242] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819257] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819307] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819311] [] ? strlcpy+0x4a/0x60 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819327] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819342] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819357] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819372] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819416] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819456] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819510] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819564] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819617] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819669] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819673] [] ? thread_return+0x4e/0x7d0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819724] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819728] [] ? kthread+0x9e/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819731] [] ? child_rip+0xa/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819733] [] ? kthread+0x0/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819736] [] ? child_rip+0x0/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819737] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819752] Call Trace: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819795] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819841] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819891] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819937] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.819984] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820000] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820010] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820022] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820034] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820084] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820095] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820110] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820139] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820155] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820184] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820198] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820213] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820264] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820267] [] ? strlcpy+0x4a/0x60 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820284] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820298] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820313] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820328] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820372] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820412] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820466] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820520] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820574] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820625] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820629] [] ? thread_return+0x4e/0x7d0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820680] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820684] [] ? kthread+0x9e/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820687] [] ? child_rip+0xa/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820690] [] ? kthread+0x0/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.820692] [] ? child_rip+0x0/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821682] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821683] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821723] CPU 7 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821724] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821760] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821761] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821764] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821768] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821770] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821771] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821773] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821775] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821776] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821778] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821780] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821781] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821785] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821787] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821788] Stack: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821789] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821791] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821793] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821796] Call Trace: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821830] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821876] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821926] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.821973] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822019] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822035] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822046] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822058] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822068] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822083] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822099] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822128] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822143] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822158] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822208] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822212] [] ? strlcpy+0x4a/0x60 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822228] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822243] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822258] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822272] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822316] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822357] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822411] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822465] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822518] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822570] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822573] [] ? thread_return+0x4e/0x7d0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822625] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822628] [] ? kthread+0x9e/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822631] [] ? child_rip+0xa/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822634] [] ? kthread+0x0/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822636] [] ? child_rip+0x0/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822637] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822653] Call Trace: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822695] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822742] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822791] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822838] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822884] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822900] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822910] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822922] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822933] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822948] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822964] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.822992] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823007] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823022] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823072] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823076] [] ? strlcpy+0x4a/0x60 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823092] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823107] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823122] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823137] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823181] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823221] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823275] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823329] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823382] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823434] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823437] [] ? thread_return+0x4e/0x7d0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823489] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823492] [] ? kthread+0x9e/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823495] [] ? child_rip+0xa/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823498] [] ? kthread+0x0/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479897.823500] [] ? child_rip+0x0/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.345965] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.443443] CPU 4 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.445574] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.543088] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.544961] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.558342] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.567062] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.572989] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.580927] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.588911] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.596876] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.604832] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.612789] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.621698] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.628068] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.636040] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.644007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.651866] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.661349] Stack: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.663759] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.671730] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.680269] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.689057] Call Trace: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.691994] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.699425] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.707540] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.715225] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.722722] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.730291] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.737939] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.745274] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.752334] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.759265] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.767205] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.774394] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.782752] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.789149] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.796142] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.804080] [] ? strlcpy+0x4a/0x60 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.809646] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.816467] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.823867] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.831096] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.838414] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.846054] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.854087] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.861503] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.868447] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.876324] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.883391] [] ? thread_return+0x4e/0x7d0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.889705] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.896694] [] ? kthread+0x9e/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.902318] [] ? child_rip+0xa/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.908091] [] ? kthread+0x0/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.913655] [] ? child_rip+0x0/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.919391] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.940823] Call Trace: Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.943734] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.951096] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.959106] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.966756] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.974290] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.981823] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.989456] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479900.996797] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.003859] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.010772] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.018652] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.025893] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.034216] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.040586] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.047633] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.055535] [] ? strlcpy+0x4a/0x60 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.061180] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.068035] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.075384] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.082576] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.089913] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.097432] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.105524] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.112933] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.119908] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.127856] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.134958] [] ? thread_return+0x4e/0x7d0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.141255] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.148106] [] ? kthread+0x9e/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.153721] [] ? child_rip+0xa/0x20 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.159416] [] ? kthread+0x0/0xc0 Aug 25 19:20:39 feral28.ccs.ornl.gov kernel: [1479901.164936] [] ? child_rip+0x0/0x20 Aug 25 19:21:07 feral28.ccs.ornl.gov kernel: [1479928.901274] Lustre: sultan-MDT000b: haven't heard from client ade025d5-35bd-119e-3200-b4438b686c43 (at 20@gni1) in 337 seconds. I think it's dead, and I am evicting it. exp ffff880be917b800, cur 1472167266 expire 1472167044 last 1472166929 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.818299] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821299] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821301] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821343] CPU 5 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821344] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821380] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821382] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821385] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821389] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821391] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821392] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821394] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821396] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821397] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821400] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821401] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821403] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821407] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821408] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821410] Stack: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821411] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821413] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821415] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821417] Call Trace: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821460] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821507] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821557] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821603] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821650] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821666] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821677] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821689] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821699] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821714] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821730] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821759] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821774] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821789] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821839] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821844] [] ? strlcpy+0x4a/0x60 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821860] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821875] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821889] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821904] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821948] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.821988] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822042] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822096] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822150] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822202] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822205] [] ? thread_return+0x4e/0x7d0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822257] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822261] [] ? kthread+0x9e/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822263] [] ? child_rip+0xa/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822266] [] ? kthread+0x0/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822268] [] ? child_rip+0x0/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822270] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822285] Call Trace: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822327] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822374] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822423] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822470] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822517] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822532] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822542] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822554] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822565] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822580] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822596] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822625] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822639] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822654] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822705] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822708] [] ? strlcpy+0x4a/0x60 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822725] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822739] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822754] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822769] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822813] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822853] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822907] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.822961] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.823015] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.823067] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.823070] [] ? thread_return+0x4e/0x7d0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.823122] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.823125] [] ? kthread+0x9e/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.823128] [] ? child_rip+0xa/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.823131] [] ? kthread+0x0/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.823133] [] ? child_rip+0x0/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825299] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825301] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825344] CPU 6 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825344] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825381] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825383] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825386] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825391] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825392] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825394] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825396] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825397] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825399] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825402] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825404] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825405] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825407] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825409] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825411] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825412] Stack: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825413] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825415] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825417] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825419] Call Trace: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825455] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825501] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825551] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825597] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825644] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825659] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825670] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825682] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825694] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825744] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825755] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825770] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825799] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825816] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825845] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825859] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825874] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825925] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825929] [] ? strlcpy+0x4a/0x60 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825945] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825960] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825975] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.825989] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826033] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826074] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826128] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826182] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826235] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826287] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826291] [] ? thread_return+0x4e/0x7d0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826343] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826346] [] ? kthread+0x9e/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826349] [] ? child_rip+0xa/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826352] [] ? kthread+0x0/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826354] [] ? child_rip+0x0/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826355] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826370] Call Trace: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826413] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826459] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826509] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826556] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826602] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826618] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826629] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826640] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826652] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826702] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826713] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826728] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826757] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826774] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826802] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826817] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826831] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826882] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826886] [] ? strlcpy+0x4a/0x60 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826902] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826916] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826931] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826946] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.826990] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827030] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827084] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827139] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827192] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827244] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827248] [] ? thread_return+0x4e/0x7d0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827299] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827303] [] ? kthread+0x9e/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827306] [] ? child_rip+0xa/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827308] [] ? kthread+0x0/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.827311] [] ? child_rip+0x0/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828299] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828300] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828341] CPU 7 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828342] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828377] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828379] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828382] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828386] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828387] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828389] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828391] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828392] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828394] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828396] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828397] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828399] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828402] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828404] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828406] Stack: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828407] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828409] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828411] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828414] Call Trace: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828447] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828494] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828543] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828590] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828637] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828652] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828663] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828675] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828686] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828701] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828717] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828746] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828760] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828775] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828825] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828829] [] ? strlcpy+0x4a/0x60 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828846] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828860] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828875] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828890] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828934] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.828974] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829028] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829082] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829136] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829187] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829191] [] ? thread_return+0x4e/0x7d0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829243] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829246] [] ? kthread+0x9e/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829249] [] ? child_rip+0xa/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829252] [] ? kthread+0x0/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829254] [] ? child_rip+0x0/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829255] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829270] Call Trace: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829313] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829359] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829409] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829455] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829502] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829518] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829528] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829540] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829551] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829566] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829582] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829610] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829625] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829640] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829690] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829694] [] ? strlcpy+0x4a/0x60 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829710] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829725] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829739] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829754] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829798] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829839] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829893] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.829947] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.830000] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.830052] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.830055] [] ? thread_return+0x4e/0x7d0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.830107] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.830111] [] ? kthread+0x9e/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.830113] [] ? child_rip+0xa/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.830116] [] ? kthread+0x0/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479981.830118] [] ? child_rip+0x0/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.352557] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.449861] CPU 4 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.452011] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.549870] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.551700] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.565124] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.573824] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.579737] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.587719] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.595736] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.603630] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.611527] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.619440] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.628376] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.634789] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.642772] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.650675] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.658649] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.668149] Stack: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.670521] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.678470] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.686973] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.695735] Call Trace: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.698638] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.706041] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.714184] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.721790] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.729322] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.736917] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.744546] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.751958] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.759100] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.766047] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.773952] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.781236] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.789541] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.795886] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.802934] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.810861] [] ? strlcpy+0x4a/0x60 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.816499] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.823327] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.830674] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.837896] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.845239] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.852870] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.860972] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.868399] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.875332] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.883244] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.890283] [] ? thread_return+0x4e/0x7d0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.896600] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.903494] [] ? kthread+0x9e/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.909126] [] ? child_rip+0xa/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.914864] [] ? kthread+0x0/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.920350] [] ? child_rip+0x0/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.926077] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.947493] Call Trace: Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.950395] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.957807] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.965882] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.973574] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.981080] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.988648] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479984.996332] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.003770] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.010867] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.017745] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.025684] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.032934] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.041214] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.047577] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.054572] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.062550] [] ? strlcpy+0x4a/0x60 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.068169] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.075007] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.082363] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.089446] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.096730] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.104318] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.112392] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.122471] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.129413] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.137282] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.144340] [] ? thread_return+0x4e/0x7d0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.150593] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.157497] [] ? kthread+0x9e/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.163173] [] ? child_rip+0xa/0x20 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.168875] [] ? kthread+0x0/0xc0 Aug 25 19:22:03 feral28.ccs.ornl.gov kernel: [1479985.174359] [] ? child_rip+0x0/0x20 Aug 25 19:22:38 feral28.ccs.ornl.gov kernel: [1480020.836492] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 19:22:38 feral28.ccs.ornl.gov kernel: [1480020.847523] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 36 previous similar messages Aug 25 19:22:38 feral28.ccs.ornl.gov kernel: [1480020.858596] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.202.11@o2ib1 (213): c: 63, oc: 0, rc: 63 Aug 25 19:22:38 feral28.ccs.ornl.gov kernel: [1480020.871938] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 36 previous similar messages Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.824916] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.827916] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.827917] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.827960] CPU 5 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.827960] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.827997] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.827999] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828002] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828006] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828008] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828009] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828011] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828013] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828014] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828017] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828019] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828020] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828024] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828026] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828027] Stack: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828028] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828030] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828032] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828034] Call Trace: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828078] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828125] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828175] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828222] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828268] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828284] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828295] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828307] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828318] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828333] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828349] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828378] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828393] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828408] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828458] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828462] [] ? strlcpy+0x4a/0x60 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828478] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828493] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828508] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828523] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828567] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828607] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828661] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828716] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828769] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828821] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828825] [] ? thread_return+0x4e/0x7d0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828877] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828880] [] ? kthread+0x9e/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828883] [] ? child_rip+0xa/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828886] [] ? kthread+0x0/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828888] [] ? child_rip+0x0/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828889] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828905] Call Trace: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828947] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.828994] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829043] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829090] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829136] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829152] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829162] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829174] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829185] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829200] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829216] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829245] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829260] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829275] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829325] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829328] [] ? strlcpy+0x4a/0x60 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829345] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829359] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829374] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829389] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829433] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829473] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829528] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829582] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829635] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829687] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829691] [] ? thread_return+0x4e/0x7d0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829742] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829746] [] ? kthread+0x9e/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829749] [] ? child_rip+0xa/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829751] [] ? kthread+0x0/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.829754] [] ? child_rip+0x0/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.831916] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.831917] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.831960] CPU 6 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.831961] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.831998] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.831999] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832002] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832007] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832009] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832011] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832012] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832014] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832016] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832018] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832020] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832021] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832027] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832028] Stack: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832029] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832031] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832033] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832036] Call Trace: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832071] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832117] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832167] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832214] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832261] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832276] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832287] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832299] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832311] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832361] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832372] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832387] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832418] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832435] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832464] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832478] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832493] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832544] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832548] [] ? strlcpy+0x4a/0x60 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832564] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832579] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832594] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832609] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832653] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832693] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832747] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832801] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832855] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832907] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832910] [] ? thread_return+0x4e/0x7d0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832962] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832965] [] ? kthread+0x9e/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832968] [] ? child_rip+0xa/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832971] [] ? kthread+0x0/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832973] [] ? child_rip+0x0/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832974] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.832989] Call Trace: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833032] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833078] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833128] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833174] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833221] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833236] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833247] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833259] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833271] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833321] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833331] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833347] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833378] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833394] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833423] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833438] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833453] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833503] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833507] [] ? strlcpy+0x4a/0x60 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833523] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833538] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833552] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833567] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833611] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833659] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833713] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833767] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833821] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833872] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833876] [] ? thread_return+0x4e/0x7d0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833928] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833931] [] ? kthread+0x9e/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833934] [] ? child_rip+0xa/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833937] [] ? kthread+0x0/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.833939] [] ? child_rip+0x0/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.834915] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.834916] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.834956] CPU 7 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.834957] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.834992] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.834994] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.834997] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835001] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835003] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835004] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835006] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835007] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835009] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835011] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835013] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835014] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835018] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835020] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835021] Stack: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835022] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835024] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835026] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835029] Call Trace: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835062] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835109] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835159] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835205] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835252] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835268] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835279] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835291] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835301] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835316] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835332] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835362] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835376] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835391] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835441] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835445] [] ? strlcpy+0x4a/0x60 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835461] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835476] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835491] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835506] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835549] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835590] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835644] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835698] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835751] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835803] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835807] [] ? thread_return+0x4e/0x7d0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835858] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835862] [] ? kthread+0x9e/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835865] [] ? child_rip+0xa/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835868] [] ? kthread+0x0/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835870] [] ? child_rip+0x0/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835871] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835886] Call Trace: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835929] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.835975] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836025] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836071] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836118] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836134] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836144] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836156] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836166] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836181] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836198] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836226] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836241] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836256] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836306] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836310] [] ? strlcpy+0x4a/0x60 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836326] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836341] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836356] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836370] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836414] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836454] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836508] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836562] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836616] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836668] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836671] [] ? thread_return+0x4e/0x7d0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836723] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836726] [] ? kthread+0x9e/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836729] [] ? child_rip+0xa/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836731] [] ? kthread+0x0/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480065.836734] [] ? child_rip+0x0/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.357415] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.454893] CPU 4 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.456999] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.554389] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.556228] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.569738] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.578458] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.584352] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.592351] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.600298] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.608169] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.616106] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.624045] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.632948] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.639309] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.647282] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.655265] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.663152] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.672659] Stack: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.675076] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.683016] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.691545] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.700384] Call Trace: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.703305] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.710744] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.718816] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.726535] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.734093] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.741679] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.749354] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.756766] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.763863] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.770853] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.778750] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.785973] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.794322] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.800676] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.807714] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.815607] [] ? strlcpy+0x4a/0x60 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.821235] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.828089] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.838108] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.845310] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.852610] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.860173] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.868221] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.875640] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.882581] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.890467] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.897604] [] ? thread_return+0x4e/0x7d0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.903901] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.910821] [] ? kthread+0x9e/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.916445] [] ? child_rip+0xa/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.922174] [] ? kthread+0x0/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.927720] [] ? child_rip+0x0/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.933457] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.954882] Call Trace: Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.957827] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.965258] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.973331] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.981013] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.988529] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480068.996081] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.003782] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.011177] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.018237] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.025098] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.032986] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.040166] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.048550] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.054913] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.061933] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.069868] [] ? strlcpy+0x4a/0x60 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.075506] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.082334] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.089676] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.096841] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.104197] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.111793] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.119885] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.127338] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.134298] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.142174] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.149282] [] ? thread_return+0x4e/0x7d0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.155582] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.162467] [] ? kthread+0x9e/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.168066] [] ? child_rip+0xa/0x20 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.173785] [] ? kthread+0x0/0xc0 Aug 25 19:23:27 feral28.ccs.ornl.gov kernel: [1480069.179340] [] ? child_rip+0x0/0x20 Aug 25 19:23:42 feral28.ccs.ornl.gov kernel: [1480084.087949] Lustre: sultan-MDT000b: haven't heard from client cc3b2647-4e02-c39c-f43e-826f0fc767f8 (at 28@gni1) in 337 seconds. I think it's dead, and I am evicting it. exp ffff880bb2d79c00, cur 1472167422 expire 1472167200 last 1472167085 Aug 25 19:23:42 feral28.ccs.ornl.gov kernel: [1480084.110806] Lustre: Skipped 1 previous similar message Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.831532] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834532] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834534] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834576] CPU 5 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834577] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834614] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834616] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834619] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834623] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834625] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834627] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834629] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834630] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834632] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834634] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834636] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834638] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834639] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834641] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834643] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834644] Stack: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834645] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834647] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834649] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834652] Call Trace: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834695] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834742] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834792] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834838] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834885] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834901] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834911] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834923] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834934] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834949] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834965] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.834995] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835010] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835025] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835075] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835079] [] ? strlcpy+0x4a/0x60 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835096] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835110] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835125] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835140] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835184] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835224] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835278] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835332] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835386] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835438] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835441] [] ? thread_return+0x4e/0x7d0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835493] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835496] [] ? kthread+0x9e/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835499] [] ? child_rip+0xa/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835502] [] ? kthread+0x0/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835504] [] ? child_rip+0x0/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835505] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835521] Call Trace: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835563] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835610] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835659] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835706] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835752] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835768] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835779] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835791] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835801] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835816] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835832] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835861] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835876] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835891] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835941] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835945] [] ? strlcpy+0x4a/0x60 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835961] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835976] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.835990] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836005] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836049] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836089] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836143] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836197] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836251] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836303] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836306] [] ? thread_return+0x4e/0x7d0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836358] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836362] [] ? kthread+0x9e/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836364] [] ? child_rip+0xa/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836367] [] ? kthread+0x0/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.836369] [] ? child_rip+0x0/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838532] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838534] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838577] CPU 6 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838578] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838615] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838617] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838620] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838625] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838626] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838628] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838629] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838631] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838633] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838635] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838637] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838639] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838640] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838642] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838644] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838646] Stack: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838647] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838649] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838651] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838653] Call Trace: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838688] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838735] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838785] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838831] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838878] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838894] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838905] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838917] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838929] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838979] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.838990] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839005] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839035] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839051] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839080] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839095] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839110] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839160] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839164] [] ? strlcpy+0x4a/0x60 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839180] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839195] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839210] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839225] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839269] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839309] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839363] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839417] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839471] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839523] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839526] [] ? thread_return+0x4e/0x7d0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839578] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839582] [] ? kthread+0x9e/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839585] [] ? child_rip+0xa/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839587] [] ? kthread+0x0/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839590] [] ? child_rip+0x0/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839591] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839606] Call Trace: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839648] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839695] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839745] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839791] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839838] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839853] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839864] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839875] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839887] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839938] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839948] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839964] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.839993] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840009] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840038] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840053] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840068] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840118] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840122] [] ? strlcpy+0x4a/0x60 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840138] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840153] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840168] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840183] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840227] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840274] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840329] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840382] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840436] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840488] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840491] [] ? thread_return+0x4e/0x7d0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840543] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840547] [] ? kthread+0x9e/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840549] [] ? child_rip+0xa/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840552] [] ? kthread+0x0/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.840554] [] ? child_rip+0x0/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841531] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841532] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841572] CPU 7 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841573] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841609] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841610] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841613] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841618] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841619] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841621] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841622] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841624] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841625] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841627] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841629] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841631] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841632] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841634] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841636] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841637] Stack: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841638] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841640] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841643] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841645] Call Trace: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841679] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841726] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841775] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841822] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841869] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841884] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841895] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841907] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841917] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841932] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841949] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841977] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.841992] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842007] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842057] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842061] [] ? strlcpy+0x4a/0x60 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842077] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842092] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842107] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842122] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842166] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842206] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842260] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842314] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842367] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842419] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842423] [] ? thread_return+0x4e/0x7d0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842474] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842478] [] ? kthread+0x9e/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842481] [] ? child_rip+0xa/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842483] [] ? kthread+0x0/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842486] [] ? child_rip+0x0/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842487] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842502] Call Trace: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842545] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842591] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842641] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842687] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842734] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842749] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842760] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842772] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842782] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842797] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842813] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842842] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842857] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842872] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842922] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842926] [] ? strlcpy+0x4a/0x60 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842942] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842957] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842971] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.842986] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843030] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843071] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843125] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843179] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843232] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843284] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843288] [] ? thread_return+0x4e/0x7d0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843339] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843343] [] ? kthread+0x9e/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843346] [] ? child_rip+0xa/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843348] [] ? kthread+0x0/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480149.843351] [] ? child_rip+0x0/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.363277] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.460530] CPU 4 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.462636] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.560329] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.562186] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.575758] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.584390] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.590319] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.598258] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.606197] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.616830] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.624761] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.632743] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.641637] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.647962] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.655876] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.663771] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.671676] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.681174] Stack: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.683584] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.691497] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.700085] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.708900] Call Trace: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.711846] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.719216] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.727297] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.734998] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.742530] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.750065] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.757695] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.765021] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.772090] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.778987] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.786805] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.794011] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.802307] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.808714] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.815813] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.823774] [] ? strlcpy+0x4a/0x60 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.829464] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.836335] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.843668] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.850888] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.858258] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.865846] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.873937] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.881339] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.888290] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.896209] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.903241] [] ? thread_return+0x4e/0x7d0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.909557] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.916477] [] ? kthread+0x9e/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.922144] [] ? child_rip+0xa/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.927838] [] ? kthread+0x0/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.933350] [] ? child_rip+0x0/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.939026] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.960485] Call Trace: Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.963404] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.970783] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.978811] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.986505] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480152.994002] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.001633] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.009357] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.016736] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.023824] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.030795] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.038675] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.045865] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.054214] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.060584] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.067526] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.075455] [] ? strlcpy+0x4a/0x60 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.081109] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.087989] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.095391] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.102568] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.109885] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.117492] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.125532] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.132897] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.139814] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.147638] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.154705] [] ? thread_return+0x4e/0x7d0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.161010] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.167932] [] ? kthread+0x9e/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.173599] [] ? child_rip+0xa/0x20 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.179302] [] ? kthread+0x0/0xc0 Aug 25 19:24:51 feral28.ccs.ornl.gov kernel: [1480153.184831] [] ? child_rip+0x0/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.838149] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841149] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841150] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841193] CPU 5 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841193] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841230] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841231] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841234] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841239] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841240] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841242] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841244] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841245] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841247] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841249] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841251] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841253] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841255] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841256] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841258] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841260] Stack: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841261] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841263] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841265] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841267] Call Trace: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841311] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841358] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841408] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841454] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841501] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841517] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841528] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841540] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841550] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841566] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841582] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841611] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841626] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841641] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841692] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841696] [] ? strlcpy+0x4a/0x60 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841712] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841727] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841742] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841756] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841800] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841841] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841895] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.841949] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842002] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842054] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842058] [] ? thread_return+0x4e/0x7d0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842109] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842113] [] ? kthread+0x9e/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842115] [] ? child_rip+0xa/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842118] [] ? kthread+0x0/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842120] [] ? child_rip+0x0/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842122] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842137] Call Trace: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842180] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842226] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842276] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842323] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842369] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842385] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842395] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842407] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842418] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842433] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842449] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842478] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842492] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842507] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842558] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842562] [] ? strlcpy+0x4a/0x60 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842578] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842592] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842607] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842622] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842666] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842706] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842760] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842814] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842868] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842920] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842923] [] ? thread_return+0x4e/0x7d0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842975] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842979] [] ? kthread+0x9e/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842981] [] ? child_rip+0xa/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842984] [] ? kthread+0x0/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.842986] [] ? child_rip+0x0/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845149] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845151] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845194] CPU 6 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845195] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845232] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845234] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845237] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845242] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845243] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845245] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845247] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845248] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845250] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845252] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845254] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845256] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845259] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845261] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845263] Stack: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845264] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845266] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845268] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845270] Call Trace: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845305] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845352] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845402] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845448] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845495] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845511] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845522] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845533] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845546] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845596] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845606] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845622] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845651] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845667] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845696] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845711] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845726] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845776] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845780] [] ? strlcpy+0x4a/0x60 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845796] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845811] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845826] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845841] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845885] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845933] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.845987] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846041] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846094] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846146] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846150] [] ? thread_return+0x4e/0x7d0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846202] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846205] [] ? kthread+0x9e/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846208] [] ? child_rip+0xa/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846210] [] ? kthread+0x0/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846213] [] ? child_rip+0x0/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846214] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846229] Call Trace: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846272] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846318] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846368] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846414] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846461] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846476] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846487] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846499] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846511] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846561] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846572] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846587] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846616] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846633] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846661] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846676] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846691] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846741] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846745] [] ? strlcpy+0x4a/0x60 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846761] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846776] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846791] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846805] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846849] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846889] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846944] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.846998] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.847051] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.847103] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.847106] [] ? thread_return+0x4e/0x7d0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.847158] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.847162] [] ? kthread+0x9e/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.847164] [] ? child_rip+0xa/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.847167] [] ? kthread+0x0/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.847169] [] ? child_rip+0x0/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848148] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848150] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848191] CPU 7 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848192] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848228] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848230] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848233] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848237] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848239] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848240] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848242] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848243] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848245] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848247] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848249] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848250] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848252] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848254] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848256] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848257] Stack: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848258] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848260] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848262] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848265] Call Trace: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848300] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848346] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848396] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848442] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848489] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848505] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848515] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848528] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848538] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848553] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848570] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848599] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848613] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848628] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848678] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848682] [] ? strlcpy+0x4a/0x60 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848699] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848713] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848728] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848743] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848787] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848827] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848882] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848936] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.848989] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849041] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849045] [] ? thread_return+0x4e/0x7d0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849096] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849100] [] ? kthread+0x9e/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849103] [] ? child_rip+0xa/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849105] [] ? kthread+0x0/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849108] [] ? child_rip+0x0/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849109] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849124] Call Trace: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849166] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849213] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849263] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849309] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849356] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849371] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849382] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849394] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849404] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849419] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849435] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849464] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849479] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849494] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849544] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849547] [] ? strlcpy+0x4a/0x60 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849564] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849578] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849593] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849608] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849652] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849692] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849746] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849800] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849853] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849905] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849908] [] ? thread_return+0x4e/0x7d0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849960] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849964] [] ? kthread+0x9e/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849967] [] ? child_rip+0xa/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849969] [] ? kthread+0x0/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480233.849971] [] ? child_rip+0x0/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.361046] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.458126] CPU 4 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.460230] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.557692] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.559546] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.572980] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.581638] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.587489] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.595350] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.603253] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.611167] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.619046] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.626914] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.635790] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.642115] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.650020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.657899] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.665742] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.675224] Stack: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.677617] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.685512] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.693919] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.702742] Call Trace: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.705626] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.713023] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.721025] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.728649] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.736084] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.743560] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.751173] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.758516] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.765586] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.772419] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.780205] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.787325] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.795562] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.801913] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.808884] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.816768] [] ? strlcpy+0x4a/0x60 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.822379] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.829215] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.836531] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.843732] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.851033] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.858613] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.866688] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.874167] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.881074] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.888949] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.896035] [] ? thread_return+0x4e/0x7d0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.902368] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.909226] [] ? kthread+0x9e/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.914842] [] ? child_rip+0xa/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.920536] [] ? kthread+0x0/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.926013] [] ? child_rip+0x0/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.931697] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.953209] Call Trace: Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.956109] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.963455] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.971501] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.979406] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.987115] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480236.994953] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.002732] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.010075] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.017111] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.024013] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.031755] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.038893] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.047144] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.053438] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.060433] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.068333] [] ? strlcpy+0x4a/0x60 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.074041] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.080904] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.088227] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.095352] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.102653] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.110259] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.118298] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.125717] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.132599] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.140526] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.147603] [] ? thread_return+0x4e/0x7d0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.153909] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.160768] [] ? kthread+0x9e/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.166373] [] ? child_rip+0xa/0x20 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.172042] [] ? kthread+0x0/0xc0 Aug 25 19:26:15 feral28.ccs.ornl.gov kernel: [1480237.177537] [] ? child_rip+0x0/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.844765] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847765] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847767] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847810] CPU 5 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847810] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847847] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847849] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847851] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847856] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847857] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847859] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847860] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847862] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847864] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847866] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847868] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847869] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847873] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847875] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847876] Stack: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847877] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847879] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847881] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847884] Call Trace: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847928] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.847975] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848024] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848071] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848118] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848133] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848144] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848156] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848167] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848182] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848198] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848228] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848242] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848257] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848308] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848312] [] ? strlcpy+0x4a/0x60 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848328] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848343] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848358] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848373] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848416] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848457] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848511] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848565] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848619] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848671] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848674] [] ? thread_return+0x4e/0x7d0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848726] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848729] [] ? kthread+0x9e/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848732] [] ? child_rip+0xa/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848735] [] ? kthread+0x0/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848737] [] ? child_rip+0x0/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848738] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848754] Call Trace: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848796] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848843] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848892] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848939] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.848985] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849001] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849012] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849024] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849034] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849049] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849065] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849094] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849109] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849123] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849174] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849177] [] ? strlcpy+0x4a/0x60 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849194] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849208] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849223] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849238] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849282] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849322] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849376] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849430] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849484] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849536] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849539] [] ? thread_return+0x4e/0x7d0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849591] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849595] [] ? kthread+0x9e/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849597] [] ? child_rip+0xa/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849600] [] ? kthread+0x0/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.849602] [] ? child_rip+0x0/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851765] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851767] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851810] CPU 6 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851810] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851847] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851849] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851852] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851857] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851858] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851860] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851862] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851863] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851865] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851867] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851869] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851871] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851874] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851877] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851878] Stack: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851879] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851881] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851883] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851885] Call Trace: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851920] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.851967] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852017] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852064] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852110] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852126] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852137] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852149] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852161] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852211] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852222] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852237] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852268] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852285] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852314] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852328] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852343] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852394] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852398] [] ? strlcpy+0x4a/0x60 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852414] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852429] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852444] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852458] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852502] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852543] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852597] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852651] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852705] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852757] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852760] [] ? thread_return+0x4e/0x7d0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852812] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852815] [] ? kthread+0x9e/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852818] [] ? child_rip+0xa/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852821] [] ? kthread+0x0/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852823] [] ? child_rip+0x0/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852825] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852840] Call Trace: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852883] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852929] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.852979] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853025] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853072] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853087] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853098] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853110] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853122] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853172] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853183] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853198] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853229] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853246] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853274] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853289] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853304] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853354] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853358] [] ? strlcpy+0x4a/0x60 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853374] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853389] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853404] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853419] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853463] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853503] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853557] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853611] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853665] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853717] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853720] [] ? thread_return+0x4e/0x7d0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853772] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853775] [] ? kthread+0x9e/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853778] [] ? child_rip+0xa/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853781] [] ? kthread+0x0/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.853783] [] ? child_rip+0x0/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854764] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854766] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854806] CPU 7 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854806] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854843] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854845] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854848] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854852] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854853] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854855] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854856] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854858] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854859] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854861] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854863] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854865] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854868] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854870] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854871] Stack: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854872] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854874] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854876] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854879] Call Trace: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854912] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.854959] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855009] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855055] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855102] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855117] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855128] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855140] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855150] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855165] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855182] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855210] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855225] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855240] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855290] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855294] [] ? strlcpy+0x4a/0x60 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855311] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855326] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855340] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855355] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855399] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855439] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855493] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855548] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855601] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855653] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855656] [] ? thread_return+0x4e/0x7d0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855708] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855711] [] ? kthread+0x9e/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855714] [] ? child_rip+0xa/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855717] [] ? kthread+0x0/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855719] [] ? child_rip+0x0/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855720] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855735] Call Trace: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855778] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855824] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855874] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855920] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855967] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855982] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.855993] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856005] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856016] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856031] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856047] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856075] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856090] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856105] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856155] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856159] [] ? strlcpy+0x4a/0x60 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856175] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856189] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856204] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856219] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856263] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856303] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856357] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856411] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856464] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856516] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856520] [] ? thread_return+0x4e/0x7d0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856572] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856575] [] ? kthread+0x9e/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856578] [] ? child_rip+0xa/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856581] [] ? kthread+0x0/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480317.856583] [] ? child_rip+0x0/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.381817] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.478973] CPU 4 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.481097] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.578696] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.580560] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.594001] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.602721] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.608648] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.616614] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.624519] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.632447] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.640388] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.648342] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.657219] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.663589] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.671493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.679398] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.687406] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.696911] Stack: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.699348] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.707287] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.715772] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.724638] Call Trace: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.727532] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.734894] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.742947] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.750615] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.758095] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.765656] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.773269] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.780690] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.787770] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.794751] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.802559] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.809793] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.818161] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.824583] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.831578] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.839583] [] ? strlcpy+0x4a/0x60 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.845177] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.852049] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.859431] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.866635] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.873926] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.881593] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.889922] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.897543] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.904826] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.913169] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.920437] [] ? thread_return+0x4e/0x7d0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.926682] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.933550] [] ? kthread+0x9e/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.939140] [] ? child_rip+0xa/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.944834] [] ? kthread+0x0/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.950300] [] ? child_rip+0x0/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.955882] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.977314] Call Trace: Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.980226] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.987649] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480320.995695] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.003345] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.010807] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.018349] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.025964] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.033298] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.040403] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.047298] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.055151] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.062314] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.070619] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.076965] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.084038] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.092000] [] ? strlcpy+0x4a/0x60 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.097637] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.104474] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.111859] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.119078] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.126310] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.133949] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.142005] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.149494] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.156497] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.164398] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.171484] [] ? thread_return+0x4e/0x7d0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.177833] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.184719] [] ? kthread+0x9e/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.192970] [] ? child_rip+0xa/0x20 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.198742] [] ? kthread+0x0/0xc0 Aug 25 19:27:39 feral28.ccs.ornl.gov kernel: [1480321.204331] [] ? child_rip+0x0/0x20 Aug 25 19:27:52 feral28.ccs.ornl.gov kernel: [1480333.988954] Lustre: sultan-MDT000b: Client 1f0727a3-d49c-044b-6def-11f5d586ab1d (at 26@gni1) reconnecting Aug 25 19:27:52 feral28.ccs.ornl.gov kernel: [1480333.999351] Lustre: Skipped 8 previous similar messages Aug 25 19:27:52 feral28.ccs.ornl.gov kernel: [1480334.005335] Lustre: sultan-MDT000b: Connection restored to c60f26de-f3d6-6cfc-b87c-120a7b1f0eed (at 26@gni1) Aug 25 19:27:52 feral28.ccs.ornl.gov kernel: [1480334.016449] Lustre: Skipped 11 previous similar messages Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.851382] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854382] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854383] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854425] CPU 5 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854426] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854462] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854464] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854467] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854472] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854473] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854475] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854477] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854479] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854480] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854482] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854484] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854486] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854488] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854490] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854492] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854493] Stack: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854494] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854496] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854499] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854501] Call Trace: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854545] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854591] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854641] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854688] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854735] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854750] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854761] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854773] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854784] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854799] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854815] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854844] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854859] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854874] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854924] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854929] [] ? strlcpy+0x4a/0x60 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854945] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854959] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854974] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.854989] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855033] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855074] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855128] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855182] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855235] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855287] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855290] [] ? thread_return+0x4e/0x7d0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855342] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855346] [] ? kthread+0x9e/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855348] [] ? child_rip+0xa/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855351] [] ? kthread+0x0/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855353] [] ? child_rip+0x0/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855355] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855370] Call Trace: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855412] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855459] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855509] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855555] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855602] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855617] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855628] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855640] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855650] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855665] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855681] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855710] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855725] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855740] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855790] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855794] [] ? strlcpy+0x4a/0x60 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855810] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855824] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855839] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855854] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855898] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855938] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.855992] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.856047] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.856100] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.856152] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.856155] [] ? thread_return+0x4e/0x7d0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.856207] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.856211] [] ? kthread+0x9e/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.856214] [] ? child_rip+0xa/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.856216] [] ? kthread+0x0/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.856218] [] ? child_rip+0x0/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858382] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858384] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858427] CPU 6 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858427] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858463] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858465] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858468] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858473] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858475] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858476] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858478] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858480] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858481] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858484] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858486] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858488] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858489] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858491] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858493] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858494] Stack: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858495] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858497] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858499] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858502] Call Trace: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858537] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858583] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858633] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858680] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858726] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858742] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858753] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858765] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858777] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858827] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858838] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858853] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858885] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858901] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858930] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858945] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.858960] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859010] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859014] [] ? strlcpy+0x4a/0x60 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859030] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859045] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859060] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859075] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859118] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859159] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859213] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859267] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859321] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859372] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859376] [] ? thread_return+0x4e/0x7d0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859428] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859431] [] ? kthread+0x9e/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859434] [] ? child_rip+0xa/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859437] [] ? kthread+0x0/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859439] [] ? child_rip+0x0/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859440] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859456] Call Trace: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859498] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859545] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859594] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859641] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859687] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859703] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859713] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859725] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859737] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859788] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859798] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859814] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859843] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859859] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859888] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859902] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859917] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859968] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859972] [] ? strlcpy+0x4a/0x60 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.859988] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860003] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860018] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860032] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860076] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860116] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860171] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860224] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860278] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860330] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860333] [] ? thread_return+0x4e/0x7d0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860385] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860388] [] ? kthread+0x9e/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860391] [] ? child_rip+0xa/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860394] [] ? kthread+0x0/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.860396] [] ? child_rip+0x0/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861381] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861382] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861422] CPU 7 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861423] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861459] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861461] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861464] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861468] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861469] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861471] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861472] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861474] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861476] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861478] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861480] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861481] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861484] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861486] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861488] Stack: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861488] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861490] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861492] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861495] Call Trace: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861528] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861575] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861625] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861671] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861718] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861733] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861744] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861756] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861766] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861781] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861797] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861826] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861841] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861856] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861906] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861910] [] ? strlcpy+0x4a/0x60 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861926] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861941] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861956] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.861971] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862015] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862055] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862109] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862163] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862216] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862268] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862272] [] ? thread_return+0x4e/0x7d0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862323] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862327] [] ? kthread+0x9e/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862329] [] ? child_rip+0xa/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862332] [] ? kthread+0x0/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862334] [] ? child_rip+0x0/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862336] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862351] Call Trace: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862393] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862440] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862489] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862536] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862582] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862598] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862609] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862620] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862631] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862646] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862662] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862691] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862706] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862721] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862771] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862775] [] ? strlcpy+0x4a/0x60 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862791] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862806] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862820] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862835] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862879] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862919] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.862973] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.863027] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.863081] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.863133] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.863136] [] ? thread_return+0x4e/0x7d0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.863188] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.863192] [] ? kthread+0x9e/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.863194] [] ? child_rip+0xa/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.863197] [] ? kthread+0x0/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480401.863200] [] ? child_rip+0x0/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.367841] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.465733] CPU 4 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.467850] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.565127] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.566992] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.580364] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.588944] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.594784] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.602708] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.610586] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.618488] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.626385] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.634265] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.643198] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.649448] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.657282] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.665127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.672989] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.682427] Stack: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.684826] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.692670] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.701139] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.709970] Call Trace: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.712864] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.720241] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.728167] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.735773] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.743209] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.750684] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.758272] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.765597] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.772651] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.779511] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.787355] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.794457] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.802754] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.809073] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.816103] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.823986] [] ? strlcpy+0x4a/0x60 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.829598] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.836460] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.843845] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.851065] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.858435] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.865988] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.874106] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.881603] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.888571] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.899150] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.906279] [] ? thread_return+0x4e/0x7d0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.912577] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.919487] [] ? kthread+0x9e/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.925069] [] ? child_rip+0xa/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.930780] [] ? kthread+0x0/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.936283] [] ? child_rip+0x0/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.942001] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.963608] Call Trace: Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.966520] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.973837] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.981892] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.989551] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480404.997004] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.004575] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.012187] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.019495] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.026522] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.033401] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.041167] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.048331] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.056541] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.062886] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.069888] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.077825] [] ? strlcpy+0x4a/0x60 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.083426] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.090247] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.097657] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.104903] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.112265] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.119895] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.127970] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.135389] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.142373] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.150320] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.157405] [] ? thread_return+0x4e/0x7d0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.163763] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.170674] [] ? kthread+0x9e/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.176280] [] ? child_rip+0xa/0x20 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.181949] [] ? kthread+0x0/0xc0 Aug 25 19:29:03 feral28.ccs.ornl.gov kernel: [1480405.187443] [] ? child_rip+0x0/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.857998] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.860998] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861000] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861043] CPU 5 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861044] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861080] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861082] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861085] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861089] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861091] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861093] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861094] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861096] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861097] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861099] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861101] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861102] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861104] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861106] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861108] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861109] Stack: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861110] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861112] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861115] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861117] Call Trace: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861161] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861208] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861258] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861304] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861351] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861367] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861378] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861390] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861401] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861416] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861432] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861461] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861476] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861491] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861541] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861545] [] ? strlcpy+0x4a/0x60 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861561] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861576] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861591] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861606] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861650] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861697] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861752] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861806] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861859] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861911] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861915] [] ? thread_return+0x4e/0x7d0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861967] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861970] [] ? kthread+0x9e/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861973] [] ? child_rip+0xa/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861976] [] ? kthread+0x0/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861978] [] ? child_rip+0x0/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861979] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.861994] Call Trace: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862037] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862083] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862133] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862180] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862226] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862242] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862252] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862264] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862275] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862290] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862306] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862335] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862350] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862365] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862415] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862419] [] ? strlcpy+0x4a/0x60 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862435] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862450] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862465] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862479] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862523] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862563] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862617] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862671] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862725] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862777] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862780] [] ? thread_return+0x4e/0x7d0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862832] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862836] [] ? kthread+0x9e/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862838] [] ? child_rip+0xa/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862841] [] ? kthread+0x0/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.862843] [] ? child_rip+0x0/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.864998] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865000] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865043] CPU 6 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865043] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865079] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865081] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865084] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865089] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865090] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865092] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865094] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865095] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865097] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865099] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865101] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865103] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865106] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865108] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865110] Stack: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865111] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865113] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865115] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865117] Call Trace: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865152] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865199] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865248] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865295] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865342] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865358] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865369] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865380] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865392] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865443] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865454] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865469] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865500] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865516] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865545] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865560] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865575] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865625] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865629] [] ? strlcpy+0x4a/0x60 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865645] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865660] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865675] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865690] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865734] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865774] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865828] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865882] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865936] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865988] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.865992] [] ? thread_return+0x4e/0x7d0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866043] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866047] [] ? kthread+0x9e/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866050] [] ? child_rip+0xa/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866052] [] ? kthread+0x0/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866055] [] ? child_rip+0x0/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866056] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866071] Call Trace: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866114] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866160] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866210] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866257] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866303] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866319] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866329] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866341] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866353] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866403] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866414] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866429] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866460] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866477] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866505] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866520] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866535] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866585] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866589] [] ? strlcpy+0x4a/0x60 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866605] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866620] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866635] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866649] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866693] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866734] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866788] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866842] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866895] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866947] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.866951] [] ? thread_return+0x4e/0x7d0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.867002] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.867006] [] ? kthread+0x9e/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.867009] [] ? child_rip+0xa/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.867011] [] ? kthread+0x0/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.867014] [] ? child_rip+0x0/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.867998] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.867999] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868039] CPU 7 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868040] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868076] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868078] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868081] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868085] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868087] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868088] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868090] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868091] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868093] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868095] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868097] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868099] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868102] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868104] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868106] Stack: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868107] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868109] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868111] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868113] Call Trace: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868147] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868194] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868243] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868290] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868337] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868352] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868363] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868375] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868385] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868400] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868417] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868446] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868460] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868475] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868526] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868529] [] ? strlcpy+0x4a/0x60 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868546] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868561] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868575] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868590] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868634] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868674] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868729] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868782] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868836] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868888] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868891] [] ? thread_return+0x4e/0x7d0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868943] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868946] [] ? kthread+0x9e/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868949] [] ? child_rip+0xa/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868952] [] ? kthread+0x0/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868954] [] ? child_rip+0x0/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868955] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.868971] Call Trace: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869013] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869060] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869109] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869156] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869202] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869218] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869228] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869240] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869251] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869266] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869282] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869311] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869325] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869340] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869390] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869394] [] ? strlcpy+0x4a/0x60 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869410] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869425] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869440] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869455] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869499] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869539] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869593] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869647] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869700] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869752] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869755] [] ? thread_return+0x4e/0x7d0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869807] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869811] [] ? kthread+0x9e/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869814] [] ? child_rip+0xa/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869816] [] ? kthread+0x0/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480485.869818] [] ? child_rip+0x0/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.398414] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.495856] CPU 4 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.498006] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.595620] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.597512] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.611059] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.619698] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.625610] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.633557] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.641495] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.649416] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.657331] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.665260] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.674163] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.680533] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.688489] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.696394] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.704417] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.716526] Stack: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.718970] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.726918] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.735427] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.744269] Call Trace: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.747206] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.754602] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.762683] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.770298] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.777819] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.785365] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.793039] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.800416] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.807435] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.814329] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.822155] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.829389] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.837746] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.844101] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.851199] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.859186] [] ? strlcpy+0x4a/0x60 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.864833] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.871653] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.879028] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.886196] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.893505] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.901042] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.909158] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.916595] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.923512] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.931422] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.938489] [] ? thread_return+0x4e/0x7d0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.944822] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.951742] [] ? kthread+0x9e/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.957384] [] ? child_rip+0xa/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.963078] [] ? kthread+0x0/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.968641] [] ? child_rip+0x0/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.974343] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.995775] Call Trace: Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480488.998705] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.006100] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.014207] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.021874] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.029319] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.036906] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.044554] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.051940] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.059045] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.065975] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.073819] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.081061] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.089445] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.095834] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.102922] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.110850] [] ? strlcpy+0x4a/0x60 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.116488] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.123315] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.130614] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.137754] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.145004] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.152547] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.160586] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.167981] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.174880] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.182702] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.189711] [] ? thread_return+0x4e/0x7d0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.195964] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.202763] [] ? kthread+0x9e/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.208327] [] ? child_rip+0xa/0x20 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.213986] [] ? kthread+0x0/0xc0 Aug 25 19:30:27 feral28.ccs.ornl.gov kernel: [1480489.219489] [] ? child_rip+0x0/0x20 Aug 25 19:30:49 feral28.ccs.ornl.gov kernel: [1480511.325424] Lustre: sultan-MDT000b: haven't heard from client b8e09222-09d9-8a4a-26f0-d58195f742d1 (at 18@gni1) in 337 seconds. I think it's dead, and I am evicting it. exp ffff880bb9cd8c00, cur 1472167849 expire 1472167627 last 1472167512 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.864615] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867615] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867616] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867658] CPU 5 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867659] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867695] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867696] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867699] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867704] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867705] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867707] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867708] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867710] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867711] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867713] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867715] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867717] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867720] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867722] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867723] Stack: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867724] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867726] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867729] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867731] Call Trace: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867775] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867821] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867871] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867918] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867965] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867981] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.867991] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868004] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868014] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868029] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868045] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868075] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868089] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868104] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868155] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868159] [] ? strlcpy+0x4a/0x60 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868175] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868190] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868205] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868219] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868263] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868304] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868358] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868412] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868466] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868518] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868521] [] ? thread_return+0x4e/0x7d0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868573] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868576] [] ? kthread+0x9e/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868579] [] ? child_rip+0xa/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868582] [] ? kthread+0x0/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868584] [] ? child_rip+0x0/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868585] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868600] Call Trace: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868643] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868689] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868739] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868786] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868832] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868848] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868858] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868870] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868881] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868896] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868912] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868941] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868956] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.868971] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869021] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869025] [] ? strlcpy+0x4a/0x60 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869041] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869056] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869071] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869085] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869129] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869169] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869223] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869278] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869331] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869383] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869387] [] ? thread_return+0x4e/0x7d0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869438] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869442] [] ? kthread+0x9e/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869444] [] ? child_rip+0xa/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869447] [] ? kthread+0x0/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.869449] [] ? child_rip+0x0/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871615] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871617] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871660] CPU 6 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871661] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871697] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871699] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871702] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871707] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871708] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871710] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871712] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871713] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871715] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871717] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871719] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871721] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871723] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871724] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871726] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871728] Stack: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871729] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871731] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871733] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871735] Call Trace: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871770] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871817] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871867] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871913] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871960] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871976] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871987] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.871998] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872011] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872061] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872072] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872087] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872118] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872135] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872164] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872178] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872193] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872244] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872248] [] ? strlcpy+0x4a/0x60 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872264] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872279] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872294] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872308] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872352] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872400] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872454] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872508] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872562] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872614] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872617] [] ? thread_return+0x4e/0x7d0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872669] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872672] [] ? kthread+0x9e/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872675] [] ? child_rip+0xa/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872678] [] ? kthread+0x0/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872680] [] ? child_rip+0x0/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872681] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872696] Call Trace: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872739] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872785] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872835] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872882] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872928] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872944] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872954] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872966] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.872978] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873028] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873039] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873054] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873083] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873100] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873128] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873143] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873158] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873208] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873212] [] ? strlcpy+0x4a/0x60 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873228] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873243] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873257] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873272] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873316] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873356] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873410] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873464] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873517] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873569] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873573] [] ? thread_return+0x4e/0x7d0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873625] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873628] [] ? kthread+0x9e/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873631] [] ? child_rip+0xa/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873634] [] ? kthread+0x0/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.873636] [] ? child_rip+0x0/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874614] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874615] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874655] CPU 7 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874655] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874692] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874694] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874697] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874701] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874702] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874704] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874706] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874707] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874709] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874711] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874712] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874714] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874719] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874721] Stack: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874721] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874723] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874725] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874728] Call Trace: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874761] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874808] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874858] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874904] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874951] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874966] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874977] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.874989] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875000] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875015] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875031] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875060] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875074] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875089] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875140] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875143] [] ? strlcpy+0x4a/0x60 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875160] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875174] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875189] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875204] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875248] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875288] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875342] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875396] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875450] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875502] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875505] [] ? thread_return+0x4e/0x7d0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875557] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875560] [] ? kthread+0x9e/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875563] [] ? child_rip+0xa/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875566] [] ? kthread+0x0/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875568] [] ? child_rip+0x0/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875569] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875585] Call Trace: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875627] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875674] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875723] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875770] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875816] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875832] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875842] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875854] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875865] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875880] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875896] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875925] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875939] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.875954] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876005] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876009] [] ? strlcpy+0x4a/0x60 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876025] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876039] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876054] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876069] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876113] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876153] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876207] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876261] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876314] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876366] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876369] [] ? thread_return+0x4e/0x7d0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876421] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876425] [] ? kthread+0x9e/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876427] [] ? child_rip+0xa/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876430] [] ? kthread+0x0/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480569.876432] [] ? child_rip+0x0/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.403115] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.503018] CPU 4 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.505134] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.602653] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.604511] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.618022] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.626707] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.632589] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.640504] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.648424] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.656365] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.664329] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.672217] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.681092] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.687401] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.695383] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.703295] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.711268] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.720752] Stack: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.723196] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.731143] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.739602] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.748434] Call Trace: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.751370] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.758749] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.766751] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.774453] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.781925] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.789520] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.797169] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.804607] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.811773] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.818719] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.826493] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.833650] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.841972] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.848369] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.855477] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.863473] [] ? strlcpy+0x4a/0x60 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.869128] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.875937] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.883217] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.890412] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.897739] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.905292] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.913315] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.920802] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.927771] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.935699] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.942818] [] ? thread_return+0x4e/0x7d0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.949047] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.955915] [] ? kthread+0x9e/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.961514] [] ? child_rip+0xa/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.967225] [] ? kthread+0x0/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.972771] [] ? child_rip+0x0/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.978462] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480572.999967] Call Trace: Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.002869] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.010263] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.018328] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.025951] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.033473] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.041089] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.048728] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.056096] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.063218] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.070148] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.077993] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.085232] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.093540] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.099859] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.106888] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.114893] [] ? strlcpy+0x4a/0x60 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.120529] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.127359] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.134732] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.141910] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.149209] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.156843] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.164872] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.172343] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.179278] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.187171] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.194264] [] ? thread_return+0x4e/0x7d0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.200606] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.207561] [] ? kthread+0x9e/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.213159] [] ? child_rip+0xa/0x20 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.218862] [] ? kthread+0x0/0xc0 Aug 25 19:31:51 feral28.ccs.ornl.gov kernel: [1480573.224433] [] ? child_rip+0x0/0x20 Aug 25 19:32:44 feral28.ccs.ornl.gov kernel: [1480626.884216] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 19:32:44 feral28.ccs.ornl.gov kernel: [1480626.895245] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 32 previous similar messages Aug 25 19:32:44 feral28.ccs.ornl.gov kernel: [1480626.906283] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.156@o2ib1 (214): c: 63, oc: 0, rc: 63 Aug 25 19:32:44 feral28.ccs.ornl.gov kernel: [1480626.919764] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 32 previous similar messages Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.871230] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874230] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874231] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874273] CPU 5 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874274] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874310] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874311] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874314] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874318] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874320] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874322] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874323] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874325] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874326] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874328] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874330] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874332] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874333] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874335] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874337] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874338] Stack: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874339] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874341] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874344] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874346] Call Trace: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874390] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874437] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874487] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874534] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874580] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874596] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874607] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874619] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874630] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874645] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874661] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874691] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874705] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874720] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874771] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874775] [] ? strlcpy+0x4a/0x60 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874791] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874806] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874821] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874835] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874879] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874920] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.874974] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875028] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875082] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875133] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875137] [] ? thread_return+0x4e/0x7d0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875188] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875192] [] ? kthread+0x9e/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875195] [] ? child_rip+0xa/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875198] [] ? kthread+0x0/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875200] [] ? child_rip+0x0/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875201] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875216] Call Trace: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875259] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875305] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875355] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875402] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875448] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875464] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875474] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875486] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875497] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875512] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875528] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875557] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875572] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875587] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875637] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875641] [] ? strlcpy+0x4a/0x60 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875657] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875672] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875687] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875701] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875745] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875786] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875840] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875894] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875947] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.875999] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.876003] [] ? thread_return+0x4e/0x7d0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.876054] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.876058] [] ? kthread+0x9e/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.876061] [] ? child_rip+0xa/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.876063] [] ? kthread+0x0/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.876066] [] ? child_rip+0x0/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878230] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878231] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878274] CPU 6 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878275] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878311] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878313] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878316] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878321] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878322] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878324] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878325] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878327] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878329] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878331] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878333] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878334] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878336] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878338] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878340] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878342] Stack: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878343] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878345] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878347] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878349] Call Trace: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878384] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878431] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878480] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878527] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878574] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878590] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878600] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878612] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878624] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878675] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878685] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878701] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878732] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878749] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878777] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878792] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878807] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878858] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878862] [] ? strlcpy+0x4a/0x60 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878878] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878893] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878907] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878922] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.878966] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879006] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879060] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879114] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879168] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879220] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879224] [] ? thread_return+0x4e/0x7d0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879276] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879279] [] ? kthread+0x9e/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879282] [] ? child_rip+0xa/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879285] [] ? kthread+0x0/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879287] [] ? child_rip+0x0/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879288] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879304] Call Trace: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879347] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879393] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879443] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879489] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879536] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879551] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879562] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879573] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879585] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879636] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879646] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879661] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879692] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879708] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879737] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879752] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879767] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879817] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879821] [] ? strlcpy+0x4a/0x60 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879837] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879851] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879866] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879881] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879925] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.879965] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880019] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880073] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880127] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880179] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880182] [] ? thread_return+0x4e/0x7d0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880234] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880237] [] ? kthread+0x9e/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880240] [] ? child_rip+0xa/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880243] [] ? kthread+0x0/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.880245] [] ? child_rip+0x0/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881229] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881230] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881270] CPU 7 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881271] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881307] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881309] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881312] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881316] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881318] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881320] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881321] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881323] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881325] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881327] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881329] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881330] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881332] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881334] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881336] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881337] Stack: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881338] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881340] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881342] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881344] Call Trace: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881378] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881424] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881474] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881520] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881567] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881583] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881593] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881605] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881616] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881631] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881647] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881676] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881691] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881706] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881756] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881760] [] ? strlcpy+0x4a/0x60 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881776] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881791] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881806] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881821] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881865] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881905] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.881959] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882013] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882067] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882119] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882122] [] ? thread_return+0x4e/0x7d0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882174] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882177] [] ? kthread+0x9e/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882180] [] ? child_rip+0xa/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882183] [] ? kthread+0x0/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882185] [] ? child_rip+0x0/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882186] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882201] Call Trace: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882244] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882291] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882340] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882387] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882433] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882449] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882459] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882471] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882481] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882497] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882513] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882541] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882556] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882571] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882621] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882625] [] ? strlcpy+0x4a/0x60 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882641] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882656] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882671] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882685] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882729] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882769] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882823] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882877] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882930] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882982] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.882985] [] ? thread_return+0x4e/0x7d0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.883037] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.883040] [] ? kthread+0x9e/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.883043] [] ? child_rip+0xa/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.883046] [] ? kthread+0x0/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480653.883048] [] ? child_rip+0x0/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.411076] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.508362] CPU 4 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.510451] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.608016] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.609895] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.623356] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.631998] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.637926] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.645864] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.653786] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.661690] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.669664] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.677558] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.686513] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.692908] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.700805] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.708683] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.716605] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.726051] Stack: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.728461] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.736432] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.744996] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.753741] Call Trace: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.756688] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.764101] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.772137] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.779797] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.787312] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.794915] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.802547] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.809941] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.817072] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.824028] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.831914] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.839148] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.847506] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.853912] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.860942] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.868911] [] ? strlcpy+0x4a/0x60 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.874540] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.881334] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.888700] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.895929] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.903298] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.910869] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.918988] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.926432] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.933365] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.941181] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.948231] [] ? thread_return+0x4e/0x7d0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.954485] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.961354] [] ? kthread+0x9e/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.966961] [] ? child_rip+0xa/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.972715] [] ? kthread+0x0/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.978262] [] ? child_rip+0x0/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480656.983997] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.005423] Call Trace: Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.008351] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.015755] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.023845] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.031495] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.038930] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.046518] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.054192] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.061527] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.068597] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.075491] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.083379] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.090541] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.098847] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.105210] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.112309] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.120271] [] ? strlcpy+0x4a/0x60 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.125926] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.132727] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.140188] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.147410] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.154718] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.162282] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.170321] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.177774] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.184699] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.192654] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.199695] [] ? thread_return+0x4e/0x7d0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.206034] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.212946] [] ? kthread+0x9e/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.218554] [] ? child_rip+0xa/0x20 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.224247] [] ? kthread+0x0/0xc0 Aug 25 19:33:15 feral28.ccs.ornl.gov kernel: [1480657.232420] [] ? child_rip+0x0/0x20 Aug 25 19:34:18 feral28.ccs.ornl.gov kernel: [1480720.391167] Lustre: sultan-MDT000b: haven't heard from client 8d48f2bb-3a94-5dec-2c5c-1f0ea8b2f1ab (at 25@gni1) in 351 seconds. I think it's dead, and I am evicting it. exp ffff880bb6759400, cur 1472168058 expire 1472167836 last 1472167707 Aug 25 19:34:18 feral28.ccs.ornl.gov kernel: [1480720.413910] Lustre: Skipped 1 previous similar message Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.877843] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880843] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880845] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880887] CPU 5 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880888] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880925] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880927] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880929] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880934] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880935] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880937] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880938] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880940] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880942] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880943] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880945] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880947] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880949] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880950] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880952] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880954] Stack: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880955] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880957] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880959] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.880961] Call Trace: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881005] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881052] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881102] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881148] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881195] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881211] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881222] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881234] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881245] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881260] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881276] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881306] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881320] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881335] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881386] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881390] [] ? strlcpy+0x4a/0x60 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881406] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881421] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881436] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881450] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881494] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881535] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881589] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881643] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881697] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881748] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881752] [] ? thread_return+0x4e/0x7d0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881803] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881807] [] ? kthread+0x9e/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881809] [] ? child_rip+0xa/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881812] [] ? kthread+0x0/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881814] [] ? child_rip+0x0/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881816] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881831] Call Trace: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881873] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881920] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.881969] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882016] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882063] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882078] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882089] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882101] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882111] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882126] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882143] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882171] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882186] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882201] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882251] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882255] [] ? strlcpy+0x4a/0x60 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882272] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882286] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882301] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882316] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882360] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882400] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882454] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882508] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882562] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882613] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882617] [] ? thread_return+0x4e/0x7d0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882669] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882672] [] ? kthread+0x9e/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882675] [] ? child_rip+0xa/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882678] [] ? kthread+0x0/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.882680] [] ? child_rip+0x0/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884843] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884845] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884888] CPU 6 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884889] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884925] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884926] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884929] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884934] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884935] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884937] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884939] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884940] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884942] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884944] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884946] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884948] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884949] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884951] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884953] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884955] Stack: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884955] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884957] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884960] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884962] Call Trace: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.884997] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885043] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885093] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885140] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885187] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885203] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885213] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885225] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885237] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885288] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885298] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885314] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885345] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885361] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885390] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885405] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885420] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885470] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885474] [] ? strlcpy+0x4a/0x60 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885491] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885505] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885520] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885535] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885579] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885619] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885673] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885727] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885781] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885833] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885837] [] ? thread_return+0x4e/0x7d0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885888] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885892] [] ? kthread+0x9e/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885895] [] ? child_rip+0xa/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885897] [] ? kthread+0x0/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885900] [] ? child_rip+0x0/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885901] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885916] Call Trace: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.885959] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886005] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886055] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886101] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886148] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886164] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886174] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886186] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886198] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886248] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886259] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886274] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886303] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886320] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886348] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886363] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886378] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886429] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886433] [] ? strlcpy+0x4a/0x60 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886449] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886464] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886478] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886493] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886537] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886577] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886631] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886685] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886739] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886791] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886795] [] ? thread_return+0x4e/0x7d0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886846] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886850] [] ? kthread+0x9e/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886852] [] ? child_rip+0xa/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886855] [] ? kthread+0x0/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.886857] [] ? child_rip+0x0/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887842] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887844] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887884] CPU 7 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887885] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887921] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887923] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887926] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887930] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887932] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887933] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887935] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887937] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887939] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887941] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887943] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887944] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887946] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887948] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887950] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887951] Stack: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887952] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887954] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887956] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887958] Call Trace: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.887998] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888044] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888094] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888140] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888187] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888203] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888214] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888226] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888236] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888251] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888267] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888296] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888311] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888326] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888376] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888380] [] ? strlcpy+0x4a/0x60 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888396] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888411] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888426] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888441] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888485] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888525] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888579] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888633] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888687] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888738] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888742] [] ? thread_return+0x4e/0x7d0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888793] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888797] [] ? kthread+0x9e/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888800] [] ? child_rip+0xa/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888803] [] ? kthread+0x0/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888805] [] ? child_rip+0x0/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888806] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888821] Call Trace: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888864] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888910] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.888960] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889006] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889053] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889069] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889079] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889091] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889102] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889117] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889133] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889162] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889176] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889191] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889241] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889245] [] ? strlcpy+0x4a/0x60 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889262] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889276] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889291] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889306] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889350] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889390] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889444] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889498] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889551] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889603] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889607] [] ? thread_return+0x4e/0x7d0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889658] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889662] [] ? kthread+0x9e/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889665] [] ? child_rip+0xa/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889667] [] ? kthread+0x0/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480737.889670] [] ? child_rip+0x0/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.409494] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.508046] CPU 4 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.510178] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.608002] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.609884] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.623326] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.631982] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.637913] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.645781] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.653686] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.661607] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.669547] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.677494] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.686404] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.692731] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.700679] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.708591] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.716548] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.726098] Stack: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.728542] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.736473] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.745070] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.753893] Call Trace: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.756840] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.764235] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.772375] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.780000] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.787532] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.795100] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.802707] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.810056] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.817179] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.824074] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.831990] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.839213] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.847553] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.853931] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.860998] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.868959] [] ? strlcpy+0x4a/0x60 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.874596] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.881379] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.888757] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.895968] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.903268] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.910866] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.918948] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.926391] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.933361] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.941151] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.948166] [] ? thread_return+0x4e/0x7d0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.954420] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.963957] [] ? kthread+0x9e/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.969580] [] ? child_rip+0xa/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.975215] [] ? kthread+0x0/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.980674] [] ? child_rip+0x0/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480740.986333] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.007819] Call Trace: Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.010721] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.018038] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.026076] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.033665] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.041101] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.048559] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.056163] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.063514] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.070568] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.077419] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.085221] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.092383] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.100646] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.106965] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.114002] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.121853] [] ? strlcpy+0x4a/0x60 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.127481] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.134344] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.141685] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.148878] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.156196] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.163750] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.171841] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.179235] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.186185] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.194079] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.201190] [] ? thread_return+0x4e/0x7d0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.207521] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.214407] [] ? kthread+0x9e/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.220030] [] ? child_rip+0xa/0x20 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.225657] [] ? kthread+0x0/0xc0 Aug 25 19:34:39 feral28.ccs.ornl.gov kernel: [1480741.231149] [] ? child_rip+0x0/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.884457] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887457] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887459] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887501] CPU 5 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887502] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887538] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887539] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887542] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887547] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887548] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887550] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887551] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887553] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887554] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887556] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887558] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887560] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887561] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887563] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887565] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887566] Stack: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887567] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887569] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887571] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887574] Call Trace: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887618] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887664] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887714] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887761] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887807] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887823] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887834] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887846] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887857] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887872] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887888] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887917] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887932] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887947] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.887997] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888001] [] ? strlcpy+0x4a/0x60 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888017] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888032] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888047] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888062] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888106] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888146] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888200] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888254] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888308] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888360] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888364] [] ? thread_return+0x4e/0x7d0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888415] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888419] [] ? kthread+0x9e/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888421] [] ? child_rip+0xa/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888424] [] ? kthread+0x0/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888427] [] ? child_rip+0x0/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888428] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888443] Call Trace: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888486] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888532] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888582] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888628] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888675] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888690] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888701] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888713] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888723] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888738] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888755] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888783] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888798] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888813] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888863] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888867] [] ? strlcpy+0x4a/0x60 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888883] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888898] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888913] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888928] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.888972] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889012] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889066] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889120] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889174] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889225] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889229] [] ? thread_return+0x4e/0x7d0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889281] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889284] [] ? kthread+0x9e/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889287] [] ? child_rip+0xa/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889290] [] ? kthread+0x0/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.889292] [] ? child_rip+0x0/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891457] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891459] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891502] CPU 6 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891503] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891540] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891542] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891544] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891549] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891551] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891552] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891554] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891556] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891558] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891560] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891562] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891564] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891567] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891569] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891571] Stack: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891572] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891574] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891576] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891579] Call Trace: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891614] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891661] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891710] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891757] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891804] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891820] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891831] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891842] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891854] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891905] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891915] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891931] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891962] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.891979] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892007] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892022] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892037] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892088] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892091] [] ? strlcpy+0x4a/0x60 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892108] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892122] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892137] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892152] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892196] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892236] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892291] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892345] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892398] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892450] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892454] [] ? thread_return+0x4e/0x7d0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892505] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892509] [] ? kthread+0x9e/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892512] [] ? child_rip+0xa/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892515] [] ? kthread+0x0/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892517] [] ? child_rip+0x0/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892518] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892533] Call Trace: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892576] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892622] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892672] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892719] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892765] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892781] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892792] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892803] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892815] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892865] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892876] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892891] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892920] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892936] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892965] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892980] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.892995] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893045] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893049] [] ? strlcpy+0x4a/0x60 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893065] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893079] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893094] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893109] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893153] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893193] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893247] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893301] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893354] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893406] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893410] [] ? thread_return+0x4e/0x7d0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893462] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893465] [] ? kthread+0x9e/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893468] [] ? child_rip+0xa/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893470] [] ? kthread+0x0/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.893473] [] ? child_rip+0x0/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894456] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894458] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894499] CPU 7 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894500] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894536] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894538] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894541] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894545] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894547] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894549] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894550] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894552] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894554] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894556] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894558] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894559] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894561] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894563] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894565] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894566] Stack: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894567] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894569] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894571] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894573] Call Trace: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894608] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894654] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894704] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894751] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894797] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894813] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894824] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894835] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894846] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894861] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894877] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894906] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894921] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894936] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894987] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.894991] [] ? strlcpy+0x4a/0x60 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895007] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895021] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895036] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895051] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895095] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895135] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895189] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895244] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895297] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895349] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895352] [] ? thread_return+0x4e/0x7d0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895404] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895408] [] ? kthread+0x9e/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895410] [] ? child_rip+0xa/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895413] [] ? kthread+0x0/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895415] [] ? child_rip+0x0/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895416] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895432] Call Trace: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895474] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895521] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895570] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895617] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895663] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895679] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895689] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895701] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895712] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895727] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895743] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895772] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895787] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895801] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895852] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895855] [] ? strlcpy+0x4a/0x60 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895872] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895886] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895901] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895916] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.895960] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896000] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896054] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896108] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896162] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896213] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896217] [] ? thread_return+0x4e/0x7d0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896268] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896272] [] ? kthread+0x9e/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896275] [] ? child_rip+0xa/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896278] [] ? kthread+0x0/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480821.896280] [] ? child_rip+0x0/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.418460] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.515868] CPU 4 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.518007] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.615736] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.617593] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.631052] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.639745] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.645708] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.653699] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.661629] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.669499] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.677447] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.685351] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.694313] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.700718] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.708699] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.716673] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.724621] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.734189] Stack: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.736633] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.747311] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.755882] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.764714] Call Trace: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.767625] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.775030] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.783136] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.790820] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.798327] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.805896] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.813536] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.820965] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.828010] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.834922] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.842758] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.849982] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.858322] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.864719] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.871817] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.879753] [] ? strlcpy+0x4a/0x60 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.885354] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.892176] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.899516] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.906693] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.914080] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.921626] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.929673] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.937075] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.944008] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.951832] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.958909] [] ? thread_return+0x4e/0x7d0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.965172] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.972092] [] ? kthread+0x9e/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.977672] [] ? child_rip+0xa/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.983436] [] ? kthread+0x0/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.988954] [] ? child_rip+0x0/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480824.994684] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.016161] Call Trace: Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.019097] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.026500] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.034531] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.042268] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.049816] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.057325] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.064991] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.072360] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.079456] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.086325] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.094161] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.101394] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.109811] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.116157] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.123281] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.131183] [] ? strlcpy+0x4a/0x60 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.136897] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.143689] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.151110] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.158250] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.165604] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.173184] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.181232] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.188581] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.195541] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.203400] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.210528] [] ? thread_return+0x4e/0x7d0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.216902] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.223874] [] ? kthread+0x9e/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.229500] [] ? child_rip+0xa/0x20 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.235237] [] ? kthread+0x0/0xc0 Aug 25 19:36:03 feral28.ccs.ornl.gov kernel: [1480825.240810] [] ? child_rip+0x0/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.891071] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894071] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894072] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894115] CPU 5 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894116] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894151] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894153] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894156] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894160] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894162] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894163] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894165] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894166] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894168] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894170] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894172] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894173] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894175] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894177] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894179] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894180] Stack: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894181] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894183] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894185] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894188] Call Trace: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894231] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894278] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894328] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894374] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894421] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894437] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894448] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894460] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894471] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894486] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894502] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894531] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894546] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894561] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894611] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894615] [] ? strlcpy+0x4a/0x60 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894632] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894646] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894661] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894676] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894720] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894760] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894815] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894869] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894922] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894974] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.894977] [] ? thread_return+0x4e/0x7d0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895029] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895033] [] ? kthread+0x9e/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895035] [] ? child_rip+0xa/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895038] [] ? kthread+0x0/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895041] [] ? child_rip+0x0/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895042] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895057] Call Trace: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895100] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895146] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895196] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895242] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895289] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895304] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895315] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895326] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895337] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895352] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895368] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895397] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895411] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895426] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895476] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895480] [] ? strlcpy+0x4a/0x60 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895496] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895511] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895526] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895541] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895585] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895625] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895679] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895733] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895787] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895839] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895842] [] ? thread_return+0x4e/0x7d0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895894] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895897] [] ? kthread+0x9e/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895900] [] ? child_rip+0xa/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895903] [] ? kthread+0x0/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.895905] [] ? child_rip+0x0/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898071] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898073] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898115] CPU 6 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898115] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898151] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898153] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898156] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898161] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898162] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898164] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898165] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898167] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898168] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898170] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898172] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898174] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898177] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898179] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898181] Stack: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898182] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898184] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898186] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898189] Call Trace: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898224] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898270] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898320] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898367] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898414] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898429] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898440] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898452] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898464] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898514] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898525] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898540] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898570] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898586] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898615] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898630] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898645] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898695] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898699] [] ? strlcpy+0x4a/0x60 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898715] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898730] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898745] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898760] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898804] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898844] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898899] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.898953] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899006] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899058] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899062] [] ? thread_return+0x4e/0x7d0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899113] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899117] [] ? kthread+0x9e/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899120] [] ? child_rip+0xa/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899122] [] ? kthread+0x0/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899125] [] ? child_rip+0x0/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899126] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899141] Call Trace: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899184] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899230] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899280] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899326] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899373] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899388] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899399] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899411] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899423] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899473] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899483] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899499] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899528] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899544] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899572] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899587] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899602] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899652] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899656] [] ? strlcpy+0x4a/0x60 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899672] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899687] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899702] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899716] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899760] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899800] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899854] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899908] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.899962] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.900014] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.900017] [] ? thread_return+0x4e/0x7d0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.900069] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.900072] [] ? kthread+0x9e/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.900075] [] ? child_rip+0xa/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.900078] [] ? kthread+0x0/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.900080] [] ? child_rip+0x0/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901070] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901072] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901111] CPU 7 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901112] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901147] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901149] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901152] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901156] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901158] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901160] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901161] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901163] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901165] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901167] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901169] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901170] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901172] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901174] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901176] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901177] Stack: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901178] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901180] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901182] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901184] Call Trace: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901218] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901264] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901314] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901360] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901407] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901423] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901434] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901446] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901456] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901471] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901487] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901516] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901531] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901546] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901596] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901600] [] ? strlcpy+0x4a/0x60 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901617] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901631] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901646] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901661] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901705] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901745] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901799] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901853] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901906] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901958] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.901962] [] ? thread_return+0x4e/0x7d0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902014] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902017] [] ? kthread+0x9e/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902020] [] ? child_rip+0xa/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902023] [] ? kthread+0x0/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902025] [] ? child_rip+0x0/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902026] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902042] Call Trace: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902084] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902130] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902180] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902227] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902273] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902289] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902300] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902312] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902322] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902337] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902353] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902382] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902397] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902412] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902462] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902466] [] ? strlcpy+0x4a/0x60 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902482] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902497] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902511] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902526] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902570] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902610] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902664] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902718] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902771] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902823] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902827] [] ? thread_return+0x4e/0x7d0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902878] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902882] [] ? kthread+0x9e/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902884] [] ? child_rip+0xa/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902887] [] ? kthread+0x0/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480905.902889] [] ? child_rip+0x0/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.424185] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.521410] CPU 4 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.523498] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.623724] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.625596] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.639048] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.647653] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.653538] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.661443] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.669433] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.677381] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.685338] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.693242] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.702221] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.708626] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.716528] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.724435] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.732426] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.742020] Stack: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.744412] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.752368] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.760869] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.769702] Call Trace: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.772585] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.779957] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.788010] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.795678] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.803132] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.810763] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.818376] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.825788] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.832910] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.839849] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.847719] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.854874] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.863126] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.869438] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.876536] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.884490] [] ? strlcpy+0x4a/0x60 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.890170] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.896963] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.904323] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.911507] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.918782] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.926327] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.934453] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.941863] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.948813] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.956760] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.963887] [] ? thread_return+0x4e/0x7d0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.970271] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.977217] [] ? kthread+0x9e/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.982816] [] ? child_rip+0xa/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.988501] [] ? kthread+0x0/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.994065] [] ? child_rip+0x0/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480908.999732] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.021252] Call Trace: Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.024180] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.031593] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.039622] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.047221] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.054743] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.062287] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.069988] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.077347] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.084442] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.091321] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.099183] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.106433] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.114816] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.121266] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.128304] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.136239] [] ? strlcpy+0x4a/0x60 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.141902] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.148713] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.156063] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.163221] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.170495] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.178095] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.186177] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.193612] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.200546] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.208474] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.215576] [] ? thread_return+0x4e/0x7d0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.221891] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.228836] [] ? kthread+0x9e/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.234451] [] ? child_rip+0xa/0x20 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.240173] [] ? kthread+0x0/0xc0 Aug 25 19:37:27 feral28.ccs.ornl.gov kernel: [1480909.245702] [] ? child_rip+0x0/0x20 Aug 25 19:38:00 feral28.ccs.ornl.gov kernel: [1480942.407854] Lustre: sultan-MDT000b: haven't heard from client 1561233c-aa4b-81d5-e0e4-de3b3609634b (at 2@gni1) in 358 seconds. I think it's dead, and I am evicting it. exp ffff880bac2d0400, cur 1472168280 expire 1472168058 last 1472167922 Aug 25 19:38:00 feral28.ccs.ornl.gov kernel: [1480942.430516] Lustre: Skipped 2 previous similar messages Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.897685] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900685] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900686] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900729] CPU 5 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900730] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900767] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900768] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900771] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900775] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900777] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900779] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900780] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900782] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900784] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900786] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900788] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900789] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900791] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900793] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900795] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900796] Stack: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900797] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900799] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900801] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900804] Call Trace: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900847] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900894] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900944] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.900990] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901037] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901053] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901063] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901076] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901086] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901101] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901117] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901147] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901161] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901176] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901227] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901231] [] ? strlcpy+0x4a/0x60 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901247] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901262] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901277] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901291] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901335] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901375] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901430] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901484] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901538] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901589] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901593] [] ? thread_return+0x4e/0x7d0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901645] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901648] [] ? kthread+0x9e/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901651] [] ? child_rip+0xa/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901654] [] ? kthread+0x0/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901656] [] ? child_rip+0x0/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901657] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901673] Call Trace: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901715] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901761] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901811] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901858] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901904] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901920] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901930] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901942] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901953] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901968] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.901984] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902012] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902027] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902042] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902092] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902096] [] ? strlcpy+0x4a/0x60 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902112] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902127] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902142] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902157] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902201] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902241] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902295] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902349] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902403] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902455] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902458] [] ? thread_return+0x4e/0x7d0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902510] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902513] [] ? kthread+0x9e/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902516] [] ? child_rip+0xa/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902519] [] ? kthread+0x0/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.902521] [] ? child_rip+0x0/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904685] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904687] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904729] CPU 6 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904730] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904767] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904769] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904772] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904776] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904778] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904779] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904781] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904782] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904784] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904786] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904788] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904789] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904791] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904793] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904795] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904797] Stack: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904798] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904800] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904802] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904804] Call Trace: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904840] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904886] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904936] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.904983] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905029] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905045] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905056] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905067] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905080] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905130] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905141] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905156] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905185] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905202] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905230] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905245] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905260] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905310] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905314] [] ? strlcpy+0x4a/0x60 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905331] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905345] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905360] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905375] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905419] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905459] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905513] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905567] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905621] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905673] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905676] [] ? thread_return+0x4e/0x7d0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905728] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905731] [] ? kthread+0x9e/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905734] [] ? child_rip+0xa/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905737] [] ? kthread+0x0/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905739] [] ? child_rip+0x0/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905740] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905756] Call Trace: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905798] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905845] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905895] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905941] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.905988] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906003] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906014] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906026] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906038] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906088] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906099] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906114] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906143] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906160] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906188] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906203] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906218] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906268] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906272] [] ? strlcpy+0x4a/0x60 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906288] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906306] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906321] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906335] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906379] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906420] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906474] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906528] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906581] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906633] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906636] [] ? thread_return+0x4e/0x7d0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906688] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906691] [] ? kthread+0x9e/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906694] [] ? child_rip+0xa/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906697] [] ? kthread+0x0/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.906699] [] ? child_rip+0x0/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907684] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907685] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907725] CPU 7 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907726] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907762] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907763] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907766] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907770] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907772] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907774] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907776] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907777] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907779] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907781] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907783] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907784] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907788] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907790] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907791] Stack: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907792] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907794] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907796] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907798] Call Trace: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907832] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907879] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907929] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.907975] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908022] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908037] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908048] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908060] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908071] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908086] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908102] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908131] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908146] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908161] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908211] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908215] [] ? strlcpy+0x4a/0x60 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908231] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908245] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908260] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908275] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908319] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908359] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908413] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908467] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908521] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908572] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908576] [] ? thread_return+0x4e/0x7d0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908628] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908631] [] ? kthread+0x9e/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908634] [] ? child_rip+0xa/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908637] [] ? kthread+0x0/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908639] [] ? child_rip+0x0/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908640] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908655] Call Trace: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908698] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908744] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908794] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908840] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908887] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908903] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908913] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908925] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908936] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908951] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908967] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.908996] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909010] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909025] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909075] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909079] [] ? strlcpy+0x4a/0x60 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909095] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909110] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909125] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909140] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909184] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909224] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909278] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909331] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909385] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909437] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909440] [] ? thread_return+0x4e/0x7d0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909492] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909495] [] ? kthread+0x9e/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909498] [] ? child_rip+0xa/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909501] [] ? kthread+0x0/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480989.909503] [] ? child_rip+0x0/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.432440] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.529682] CPU 4 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.531763] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.629379] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.631243] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.644796] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.653509] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.659384] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.667339] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.675323] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.683235] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.691115] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.699062] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.707963] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.714368] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.722367] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.730331] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.738278] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.747804] Stack: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.750178] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.758135] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.766672] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.775590] Call Trace: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.778545] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.785976] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.794099] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.801775] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.809255] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.816852] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.824508] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.831850] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.838930] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.845851] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.853695] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.860831] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.869104] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.875492] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.882573] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.890517] [] ? strlcpy+0x4a/0x60 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.896145] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.902974] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.910411] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.917622] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.924913] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.932563] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.940645] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.948047] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.954963] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.962813] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.969924] [] ? thread_return+0x4e/0x7d0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.976176] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.983080] [] ? kthread+0x9e/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.988757] [] ? child_rip+0xa/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.994486] [] ? kthread+0x0/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480992.999971] [] ? child_rip+0x0/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.005681] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.027148] Call Trace: Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.030076] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.037543] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.045622] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.053326] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.060858] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.068436] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.076032] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.083418] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.090461] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.097401] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.105237] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.112418] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.120715] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.127059] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.134114] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.142041] [] ? strlcpy+0x4a/0x60 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.147714] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.154499] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.161883] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.169051] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.176369] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.183982] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.192091] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.199510] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.206496] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.214371] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.221483] [] ? thread_return+0x4e/0x7d0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.227824] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.234691] [] ? kthread+0x9e/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.240290] [] ? child_rip+0xa/0x20 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.245975] [] ? kthread+0x0/0xc0 Aug 25 19:38:51 feral28.ccs.ornl.gov kernel: [1480993.251477] [] ? child_rip+0x0/0x20 Aug 25 19:38:55 feral28.ccs.ornl.gov kernel: [1480997.175573] Lustre: sultan-MDT000b: Client 5d9f30ae-edef-9a3d-c88a-aa425ee6b9cf (at 15@gni1) reconnecting Aug 25 19:38:55 feral28.ccs.ornl.gov kernel: [1480997.185930] Lustre: Skipped 4 previous similar messages Aug 25 19:38:55 feral28.ccs.ornl.gov kernel: [1480997.191723] Lustre: sultan-MDT000b: Connection restored to 62645a1e-aa58-fbd2-2d3b-02416b8b730f (at 15@gni1) Aug 25 19:38:55 feral28.ccs.ornl.gov kernel: [1480997.202418] Lustre: Skipped 9 previous similar messages Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.904299] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907299] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907300] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907343] CPU 5 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907344] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907379] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907381] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907384] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907388] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907390] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907392] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907393] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907395] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907396] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907398] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907400] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907402] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907403] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907405] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907407] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907408] Stack: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907409] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907411] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907414] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907416] Call Trace: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907460] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907506] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907556] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907603] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907650] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907665] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907676] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907688] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907699] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907714] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907730] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907759] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907774] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907789] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907839] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907843] [] ? strlcpy+0x4a/0x60 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907859] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907874] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907889] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907904] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907948] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.907988] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908042] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908097] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908150] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908202] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908205] [] ? thread_return+0x4e/0x7d0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908257] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908261] [] ? kthread+0x9e/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908264] [] ? child_rip+0xa/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908266] [] ? kthread+0x0/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908269] [] ? child_rip+0x0/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908270] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908285] Call Trace: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908328] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908374] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908424] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908470] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908517] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908532] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908543] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908555] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908565] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908580] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908596] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908625] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908640] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908655] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908705] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908709] [] ? strlcpy+0x4a/0x60 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908725] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908739] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908754] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908769] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908813] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908853] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908908] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.908962] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.909016] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.909067] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.909071] [] ? thread_return+0x4e/0x7d0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.909123] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.909126] [] ? kthread+0x9e/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.909129] [] ? child_rip+0xa/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.909132] [] ? kthread+0x0/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.909134] [] ? child_rip+0x0/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911299] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911301] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911343] CPU 6 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911344] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911380] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911382] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911385] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911389] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911391] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911392] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911394] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911395] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911397] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911399] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911401] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911402] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911404] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911406] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911408] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911409] Stack: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911410] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911412] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911414] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911417] Call Trace: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911452] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911498] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911548] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911595] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911642] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911657] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911668] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911680] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911692] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911742] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911753] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911768] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911797] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911814] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911843] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911857] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911872] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911923] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911927] [] ? strlcpy+0x4a/0x60 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911943] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911958] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911973] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.911987] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912031] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912072] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912126] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912180] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912233] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912286] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912289] [] ? thread_return+0x4e/0x7d0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912341] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912344] [] ? kthread+0x9e/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912347] [] ? child_rip+0xa/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912350] [] ? kthread+0x0/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912352] [] ? child_rip+0x0/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912353] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912368] Call Trace: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912411] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912457] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912507] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912553] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912600] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912615] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912626] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912637] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912650] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912700] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912710] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912725] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912754] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912771] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912799] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912814] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912829] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912879] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912883] [] ? strlcpy+0x4a/0x60 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912899] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912913] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912928] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912943] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.912987] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913027] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913082] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913136] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913189] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913241] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913245] [] ? thread_return+0x4e/0x7d0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913296] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913300] [] ? kthread+0x9e/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913302] [] ? child_rip+0xa/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913305] [] ? kthread+0x0/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.913307] [] ? child_rip+0x0/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914298] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914300] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914339] CPU 7 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914340] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914377] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914379] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914382] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914386] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914388] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914389] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914391] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914393] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914394] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914396] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914398] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914400] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914402] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914403] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914405] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914407] Stack: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914407] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914409] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914411] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914414] Call Trace: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914448] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914494] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914544] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914591] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914637] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914653] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914664] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914676] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914686] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914701] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914717] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914746] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914761] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914776] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914826] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914830] [] ? strlcpy+0x4a/0x60 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914846] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914861] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914875] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914890] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914934] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.914974] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915028] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915082] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915136] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915188] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915191] [] ? thread_return+0x4e/0x7d0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915243] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915246] [] ? kthread+0x9e/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915249] [] ? child_rip+0xa/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915252] [] ? kthread+0x0/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915254] [] ? child_rip+0x0/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915255] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915271] Call Trace: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915313] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915359] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915409] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915455] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915502] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915518] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915528] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915540] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915550] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915566] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915581] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915610] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915625] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915640] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915690] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915694] [] ? strlcpy+0x4a/0x60 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915710] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915725] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915739] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915754] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915798] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915838] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915892] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.915946] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.916000] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.916051] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.916055] [] ? thread_return+0x4e/0x7d0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.916107] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.916110] [] ? kthread+0x9e/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.916113] [] ? child_rip+0xa/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.916116] [] ? kthread+0x0/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481073.916118] [] ? child_rip+0x0/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.439395] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.536846] CPU 4 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.538969] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.636742] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.638597] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.652082] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.660741] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.666660] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.674573] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.682477] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.690434] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.698433] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.706416] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.715343] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.721721] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.729625] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.737529] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.745470] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.754976] Stack: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.757386] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.765334] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.773844] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.782736] Call Trace: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.785656] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.793070] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.801142] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.808835] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.816271] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.823780] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.831428] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.838840] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.845893] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.852780] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.860650] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.867874] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.876163] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.882577] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.889607] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.897514] [] ? strlcpy+0x4a/0x60 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.903117] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.909982] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.917357] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.924516] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.931860] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.939413] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.947547] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.954993] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.961996] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.969925] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.977069] [] ? thread_return+0x4e/0x7d0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.983359] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.990183] [] ? kthread+0x9e/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481076.995753] [] ? child_rip+0xa/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.001492] [] ? kthread+0x0/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.007039] [] ? child_rip+0x0/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.012793] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.034287] Call Trace: Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.037241] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.047324] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.055370] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.063019] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.070525] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.078069] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.085672] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.092990] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.100129] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.107058] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.114939] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.122111] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.130363] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.136744] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.143817] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.151744] [] ? strlcpy+0x4a/0x60 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.157382] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.164270] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.171672] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.178883] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.186175] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.193711] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.201795] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.209213] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.216146] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.224015] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.231125] [] ? thread_return+0x4e/0x7d0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.237388] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.244316] [] ? kthread+0x9e/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.249975] [] ? child_rip+0xa/0x20 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.255677] [] ? kthread+0x0/0xc0 Aug 25 19:40:15 feral28.ccs.ornl.gov kernel: [1481077.261250] [] ? child_rip+0x0/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.910913] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.913913] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.913914] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.913958] CPU 5 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.913959] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.913995] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.913997] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914000] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914005] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914006] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914008] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914009] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914011] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914012] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914014] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914016] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914018] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914021] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914023] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914024] Stack: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914025] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914027] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914030] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914032] Call Trace: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914078] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914125] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914175] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914222] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914268] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914285] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914296] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914309] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914319] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914335] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914351] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914381] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914396] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914411] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914461] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914466] [] ? strlcpy+0x4a/0x60 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914482] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914497] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914512] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914527] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914571] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914611] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914665] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914719] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914773] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914825] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914828] [] ? thread_return+0x4e/0x7d0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914880] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914884] [] ? kthread+0x9e/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914887] [] ? child_rip+0xa/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914890] [] ? kthread+0x0/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914892] [] ? child_rip+0x0/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914893] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914908] Call Trace: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914951] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.914997] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915047] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915094] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915140] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915156] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915166] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915178] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915189] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915204] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915220] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915249] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915263] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915278] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915328] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915332] [] ? strlcpy+0x4a/0x60 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915348] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915363] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915378] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915393] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915437] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915477] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915531] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915585] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915639] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915691] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915694] [] ? thread_return+0x4e/0x7d0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915746] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915750] [] ? kthread+0x9e/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915753] [] ? child_rip+0xa/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915755] [] ? kthread+0x0/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.915757] [] ? child_rip+0x0/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.917913] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.917915] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.917957] CPU 6 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.917958] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.917994] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.917996] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.917999] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918004] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918006] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918007] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918009] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918010] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918012] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918015] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918016] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918018] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918024] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918025] Stack: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918026] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918028] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918031] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918033] Call Trace: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918068] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918115] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918165] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918212] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918259] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918275] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918285] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918297] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918309] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918360] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918370] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918386] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918417] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918434] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918462] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918477] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918492] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918542] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918547] [] ? strlcpy+0x4a/0x60 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918563] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918578] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918593] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918607] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918651] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918692] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918746] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918800] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918854] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918906] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918909] [] ? thread_return+0x4e/0x7d0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918961] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918964] [] ? kthread+0x9e/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918967] [] ? child_rip+0xa/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918970] [] ? kthread+0x0/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918972] [] ? child_rip+0x0/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918974] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.918989] Call Trace: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919032] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919078] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919128] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919174] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919221] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919236] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919247] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919258] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919270] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919321] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919331] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919347] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919375] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919392] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919420] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919435] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919450] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919500] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919504] [] ? strlcpy+0x4a/0x60 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919520] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919535] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919550] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919564] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919608] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919648] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919702] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919756] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919810] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919862] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919865] [] ? thread_return+0x4e/0x7d0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919917] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919921] [] ? kthread+0x9e/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919923] [] ? child_rip+0xa/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919926] [] ? kthread+0x0/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.919928] [] ? child_rip+0x0/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.920912] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.920913] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.920954] CPU 7 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.920955] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.920991] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.920993] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.920996] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921000] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921002] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921003] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921005] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921006] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921008] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921010] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921012] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921014] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921018] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921020] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921021] Stack: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921022] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921024] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921026] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921029] Call Trace: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921062] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921109] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921159] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921205] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921252] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921268] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921278] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921290] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921301] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921316] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921332] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921361] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921376] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921391] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921441] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921445] [] ? strlcpy+0x4a/0x60 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921461] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921476] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921491] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921506] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921550] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921590] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921644] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921698] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921751] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921803] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921806] [] ? thread_return+0x4e/0x7d0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921858] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921861] [] ? kthread+0x9e/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921864] [] ? child_rip+0xa/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921867] [] ? kthread+0x0/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921869] [] ? child_rip+0x0/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921870] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921886] Call Trace: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921928] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.921975] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922024] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922071] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922118] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922133] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922144] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922156] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922166] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922181] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922197] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922226] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922241] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922256] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922306] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922310] [] ? strlcpy+0x4a/0x60 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922326] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922341] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922355] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922370] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922414] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922454] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922508] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922562] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922616] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922668] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922671] [] ? thread_return+0x4e/0x7d0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922723] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922726] [] ? kthread+0x9e/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922729] [] ? child_rip+0xa/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922732] [] ? kthread+0x0/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481157.922734] [] ? child_rip+0x0/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.443195] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.540352] CPU 4 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.542492] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.640091] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.641973] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.655536] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.664220] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.670140] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.678096] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.686094] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.694060] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.701965] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.709895] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.718771] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.725123] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.733097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.741009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.748939] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.758412] Stack: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.760831] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.768718] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.777185] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.786026] Call Trace: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.788980] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.796367] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.807185] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.814889] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.822334] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.829921] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.837578] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.844981] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.852111] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.859094] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.866946] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.874179] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.882571] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.888933] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.895989] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.904003] [] ? strlcpy+0x4a/0x60 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.909606] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.916451] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.923791] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.930932] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.938200] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.945770] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.953819] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.961134] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.968154] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.976056] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.983184] [] ? thread_return+0x4e/0x7d0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.989550] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481160.996515] [] ? kthread+0x9e/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.002181] [] ? child_rip+0xa/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.007954] [] ? kthread+0x0/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.013507] [] ? child_rip+0x0/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.019220] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.040592] Call Trace: Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.043494] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.050925] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.059075] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.066751] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.074231] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.081740] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.089458] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.096842] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.103974] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.110886] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.118783] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.125980] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.134311] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.140657] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.147704] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.155718] [] ? strlcpy+0x4a/0x60 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.161364] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.168236] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.175663] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.182891] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.190166] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.197771] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.205854] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.213222] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.220224] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.228101] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.235263] [] ? thread_return+0x4e/0x7d0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.241612] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.248532] [] ? kthread+0x9e/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.254165] [] ? child_rip+0xa/0x20 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.259886] [] ? kthread+0x0/0xc0 Aug 25 19:41:39 feral28.ccs.ornl.gov kernel: [1481161.265432] [] ? child_rip+0x0/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.917527] BUG: soft lockup - CPU#4 stuck for 68s! [mdt01_002:21487] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920527] BUG: soft lockup - CPU#5 stuck for 68s! [mdt01_007:17158] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920528] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920570] CPU 5 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920571] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920608] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920609] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920612] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920617] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920618] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920620] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920621] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920623] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920625] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920627] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920629] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920630] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920632] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920634] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920636] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920637] Stack: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920638] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920640] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920642] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920645] Call Trace: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920689] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920735] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920785] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920832] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920878] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920894] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920905] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920917] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920928] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920943] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920959] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.920988] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921003] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921018] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921068] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921073] [] ? strlcpy+0x4a/0x60 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921089] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921104] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921118] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921133] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921177] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921217] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921271] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921325] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921379] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921431] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921434] [] ? thread_return+0x4e/0x7d0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921486] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921489] [] ? kthread+0x9e/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921492] [] ? child_rip+0xa/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921495] [] ? kthread+0x0/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921497] [] ? child_rip+0x0/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921498] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921513] Call Trace: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921556] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921603] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921652] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921699] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921745] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921761] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921771] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921783] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921794] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921809] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921825] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921854] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921868] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921883] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921933] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921937] [] ? strlcpy+0x4a/0x60 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921953] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921968] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921983] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.921998] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922042] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922082] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922136] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922190] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922243] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922295] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922299] [] ? thread_return+0x4e/0x7d0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922351] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922354] [] ? kthread+0x9e/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922357] [] ? child_rip+0xa/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922359] [] ? kthread+0x0/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.922362] [] ? child_rip+0x0/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924527] BUG: soft lockup - CPU#6 stuck for 68s! [mdt01_000:21485] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924529] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924571] CPU 6 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924572] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924608] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924610] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924613] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924617] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924619] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924620] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924622] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924623] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924625] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924627] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924629] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924630] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924632] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924634] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924636] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924637] Stack: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924638] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924640] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924642] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924645] Call Trace: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924680] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924726] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924776] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924823] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924870] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924886] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924897] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924908] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924921] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924971] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924982] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.924997] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925028] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925045] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925073] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925088] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925103] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925154] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925158] [] ? strlcpy+0x4a/0x60 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925174] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925189] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925203] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925218] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925262] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925302] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925357] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925410] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925464] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925516] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925520] [] ? thread_return+0x4e/0x7d0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925571] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925575] [] ? kthread+0x9e/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925578] [] ? child_rip+0xa/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925580] [] ? kthread+0x0/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925583] [] ? child_rip+0x0/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925584] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925599] Call Trace: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925642] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925688] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925738] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925784] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925831] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925847] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925857] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925869] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925881] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925931] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925942] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925957] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.925986] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926003] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926031] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926046] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926061] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926111] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926115] [] ? strlcpy+0x4a/0x60 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926131] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926146] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926161] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926176] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926219] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926260] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926314] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926367] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926421] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926473] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926477] [] ? thread_return+0x4e/0x7d0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926529] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926532] [] ? kthread+0x9e/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926535] [] ? child_rip+0xa/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926538] [] ? kthread+0x0/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.926540] [] ? child_rip+0x0/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927526] BUG: soft lockup - CPU#7 stuck for 68s! [mdt01_001:21486] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927528] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927567] CPU 7 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927568] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927604] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927606] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927609] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927613] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927615] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927617] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927618] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927620] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927622] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927624] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927626] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927628] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927631] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927633] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927635] Stack: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927636] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927638] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927640] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927642] Call Trace: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927676] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927723] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927772] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927819] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927866] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927881] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927892] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927904] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927915] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927930] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927946] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927975] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.927989] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928004] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928055] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928059] [] ? strlcpy+0x4a/0x60 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928075] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928090] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928104] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928119] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928163] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928204] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928258] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928311] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928365] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928417] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928420] [] ? thread_return+0x4e/0x7d0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928472] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928475] [] ? kthread+0x9e/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928478] [] ? child_rip+0xa/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928481] [] ? kthread+0x0/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928483] [] ? child_rip+0x0/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928484] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928499] Call Trace: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928542] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928588] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928638] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928684] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928731] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928746] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928757] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928769] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928780] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928795] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928811] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928839] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928854] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928869] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928919] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928923] [] ? strlcpy+0x4a/0x60 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928939] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928954] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928968] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.928983] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929027] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929067] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929121] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929175] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929229] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929281] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929284] [] ? thread_return+0x4e/0x7d0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929336] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929339] [] ? kthread+0x9e/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929342] [] ? child_rip+0xa/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929345] [] ? kthread+0x0/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481241.929347] [] ? child_rip+0x0/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.450410] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.547905] CPU 4 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.550011] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.650288] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.652169] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.665654] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.674348] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.680239] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.688221] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.696135] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.704073] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.711988] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.719927] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.728898] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.735198] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.743181] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.751145] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.759143] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.768600] Stack: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.771002] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.778906] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.787389] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.796153] Call Trace: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.799055] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.806468] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.814556] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.822251] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.829800] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.837344] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.844914] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.852308] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.859440] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.866361] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.874272] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.881447] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.889734] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.896097] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.903134] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.911080] [] ? strlcpy+0x4a/0x60 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.916674] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.923458] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.930867] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.938096] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.945370] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.952959] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.961017] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.968392] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.975360] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.983305] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.990391] [] ? thread_return+0x4e/0x7d0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481244.996714] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.003642] [] ? kthread+0x9e/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.009249] [] ? child_rip+0xa/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.014987] [] ? kthread+0x0/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.020489] [] ? child_rip+0x0/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.026242] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.047729] Call Trace: Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.050639] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.058053] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.066143] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.073896] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.081445] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.088972] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.096612] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.104006] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.111112] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.118023] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.125920] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.133144] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.141448] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.147838] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.154841] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.162804] [] ? strlcpy+0x4a/0x60 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.168450] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.175233] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.182644] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.189864] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.197216] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.204821] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.212905] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.220324] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.227301] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.235220] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.242236] [] ? thread_return+0x4e/0x7d0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.248524] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.255418] [] ? kthread+0x9e/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.261060] [] ? child_rip+0xa/0x20 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.266745] [] ? kthread+0x0/0xc0 Aug 25 19:43:03 feral28.ccs.ornl.gov kernel: [1481245.272246] [] ? child_rip+0x0/0x20 Aug 25 19:43:42 feral28.ccs.ornl.gov kernel: [1481284.936034] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 19:43:42 feral28.ccs.ornl.gov kernel: [1481284.947006] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 34 previous similar messages Aug 25 19:43:42 feral28.ccs.ornl.gov kernel: [1481284.958137] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.159@o2ib1 (212): c: 63, oc: 0, rc: 63 Aug 25 19:43:42 feral28.ccs.ornl.gov kernel: [1481284.971558] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 34 previous similar messages Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.924141] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927141] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927142] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927185] CPU 5 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927186] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927222] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927224] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927227] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927231] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927233] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927234] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927236] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927237] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927239] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927241] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927243] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927245] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927246] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927248] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927250] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927252] Stack: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927253] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927255] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927257] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927259] Call Trace: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927303] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927349] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927399] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927446] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927492] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927508] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927519] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927531] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927542] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927557] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927573] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927602] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927617] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927632] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927683] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927687] [] ? strlcpy+0x4a/0x60 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927703] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927718] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927733] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927748] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927791] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927832] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927886] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927940] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.927993] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928045] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928049] [] ? thread_return+0x4e/0x7d0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928100] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928104] [] ? kthread+0x9e/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928107] [] ? child_rip+0xa/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928109] [] ? kthread+0x0/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928112] [] ? child_rip+0x0/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928113] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928128] Call Trace: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928171] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928217] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928267] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928313] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928360] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928375] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928386] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928398] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928409] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928424] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928440] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928468] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928483] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928498] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928548] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928552] [] ? strlcpy+0x4a/0x60 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928568] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928583] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928598] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928612] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928656] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928697] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928751] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928805] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928858] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928910] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928914] [] ? thread_return+0x4e/0x7d0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928965] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928969] [] ? kthread+0x9e/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928972] [] ? child_rip+0xa/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928974] [] ? kthread+0x0/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.928977] [] ? child_rip+0x0/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931141] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931143] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931186] CPU 6 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931186] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931223] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931225] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931228] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931233] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931234] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931236] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931238] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931239] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931241] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931243] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931245] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931247] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931251] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931252] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931254] Stack: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931255] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931257] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931259] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931261] Call Trace: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931296] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931342] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931392] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931438] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931485] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931501] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931511] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931523] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931535] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931586] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931596] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931612] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931643] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931659] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931688] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931703] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931718] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931768] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931772] [] ? strlcpy+0x4a/0x60 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931789] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931803] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931818] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931833] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931877] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931917] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.931971] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932025] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932079] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932131] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932135] [] ? thread_return+0x4e/0x7d0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932187] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932190] [] ? kthread+0x9e/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932193] [] ? child_rip+0xa/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932196] [] ? kthread+0x0/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932198] [] ? child_rip+0x0/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932199] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932214] Call Trace: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932257] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932303] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932353] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932399] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932446] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932462] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932472] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932484] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932496] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932546] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932557] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932572] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932603] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932619] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932648] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932663] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932678] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932728] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932732] [] ? strlcpy+0x4a/0x60 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932748] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932763] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932778] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932792] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932836] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932876] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932930] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.932984] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.933038] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.933090] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.933093] [] ? thread_return+0x4e/0x7d0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.933145] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.933149] [] ? kthread+0x9e/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.933151] [] ? child_rip+0xa/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.933154] [] ? kthread+0x0/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.933156] [] ? child_rip+0x0/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934140] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934142] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934181] CPU 7 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934182] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934217] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934219] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934222] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934226] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934228] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934229] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934231] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934232] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934234] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934236] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934238] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934239] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934241] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934243] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934245] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934246] Stack: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934247] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934249] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934252] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934254] Call Trace: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934288] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934335] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934384] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934431] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934478] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934493] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934504] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934516] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934527] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934542] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934558] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934587] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934602] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934616] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934667] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934671] [] ? strlcpy+0x4a/0x60 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934687] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934702] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934716] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934731] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934775] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934815] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934869] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934923] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.934976] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935028] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935031] [] ? thread_return+0x4e/0x7d0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935083] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935087] [] ? kthread+0x9e/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935089] [] ? child_rip+0xa/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935092] [] ? kthread+0x0/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935094] [] ? child_rip+0x0/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935095] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935111] Call Trace: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935153] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935200] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935249] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935296] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935343] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935358] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935369] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935381] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935391] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935406] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935422] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935451] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935466] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935481] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935531] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935535] [] ? strlcpy+0x4a/0x60 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935551] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935566] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935580] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935595] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935639] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935679] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935733] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935787] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935841] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935892] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935896] [] ? thread_return+0x4e/0x7d0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935948] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935951] [] ? kthread+0x9e/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935954] [] ? child_rip+0xa/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935957] [] ? kthread+0x0/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481325.935959] [] ? child_rip+0x0/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.459619] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.556827] CPU 4 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.558951] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.656661] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.658509] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.672072] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.680670] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.686529] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.694501] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.702396] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.710311] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.718241] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.726163] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.735149] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.741528] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.749417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.757381] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.765347] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.774870] Stack: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.777307] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.785271] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.793869] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.802734] Call Trace: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.805602] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.813016] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.821149] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.828842] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.836330] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.843924] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.851609] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.858994] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.866075] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.872951] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.880769] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.887933] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.896256] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.902610] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.909682] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.917662] [] ? strlcpy+0x4a/0x60 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.923308] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.930110] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.937520] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.944713] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.952050] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.959629] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.967660] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.975113] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.982029] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.989862] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481328.996982] [] ? thread_return+0x4e/0x7d0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.003296] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.010233] [] ? kthread+0x9e/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.015823] [] ? child_rip+0xa/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.021596] [] ? kthread+0x0/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.027141] [] ? child_rip+0x0/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.032878] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.054311] Call Trace: Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.057230] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.064601] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.072708] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.080409] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.087924] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.095416] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.103107] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.110493] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.117589] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.124528] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.132424] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.139613] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.147945] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.154343] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.161355] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.169307] [] ? strlcpy+0x4a/0x60 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.174874] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.181728] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.189131] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.196349] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.203704] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.211239] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.219304] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.226741] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.233649] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.241552] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.248636] [] ? thread_return+0x4e/0x7d0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.254933] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.261819] [] ? kthread+0x9e/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.267495] [] ? child_rip+0xa/0x20 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.273267] [] ? kthread+0x0/0xc0 Aug 25 19:44:27 feral28.ccs.ornl.gov kernel: [1481329.278778] [] ? child_rip+0x0/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.930755] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933755] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933757] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933799] CPU 5 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933800] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933837] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933838] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933841] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933846] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933847] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933849] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933851] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933852] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933854] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933856] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933858] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933860] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933861] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933863] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933865] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933866] Stack: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933867] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933869] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933871] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933874] Call Trace: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933917] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.933963] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934013] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934060] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934106] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934122] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934133] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934145] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934156] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934171] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934187] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934216] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934231] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934246] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934296] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934300] [] ? strlcpy+0x4a/0x60 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934316] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934331] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934346] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934361] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934404] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934445] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934499] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934553] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934606] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934658] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934661] [] ? thread_return+0x4e/0x7d0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934713] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934717] [] ? kthread+0x9e/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934720] [] ? child_rip+0xa/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934722] [] ? kthread+0x0/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934725] [] ? child_rip+0x0/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934726] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934741] Call Trace: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934783] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934830] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934879] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934926] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934972] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934988] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.934998] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935010] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935021] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935036] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935052] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935081] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935095] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935110] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935160] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935164] [] ? strlcpy+0x4a/0x60 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935181] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935195] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935210] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935225] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935269] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935309] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935363] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935417] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935471] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935522] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935526] [] ? thread_return+0x4e/0x7d0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935577] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935581] [] ? kthread+0x9e/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935584] [] ? child_rip+0xa/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935586] [] ? kthread+0x0/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.935588] [] ? child_rip+0x0/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937755] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937757] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937799] CPU 6 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937800] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937836] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937838] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937841] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937845] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937847] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937849] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937850] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937852] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937853] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937855] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937857] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937859] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937860] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937862] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937864] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937865] Stack: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937866] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937868] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937870] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937873] Call Trace: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937908] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.937955] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938005] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938051] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938098] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938114] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938125] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938136] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938149] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938199] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938210] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938225] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938257] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938273] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938302] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938316] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938331] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938382] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938386] [] ? strlcpy+0x4a/0x60 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938402] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938417] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938432] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938447] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938490] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938531] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938585] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938639] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938692] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938744] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938748] [] ? thread_return+0x4e/0x7d0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938799] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938803] [] ? kthread+0x9e/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938806] [] ? child_rip+0xa/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938808] [] ? kthread+0x0/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938811] [] ? child_rip+0x0/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938812] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938827] Call Trace: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938870] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938916] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.938966] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939012] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939059] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939075] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939085] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939097] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939109] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939159] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939170] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939185] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939216] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939232] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939261] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939275] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939290] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939341] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939345] [] ? strlcpy+0x4a/0x60 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939361] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939376] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939391] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939405] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939449] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939489] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939543] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939597] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939651] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939703] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939706] [] ? thread_return+0x4e/0x7d0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939758] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939762] [] ? kthread+0x9e/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939765] [] ? child_rip+0xa/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939767] [] ? kthread+0x0/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.939770] [] ? child_rip+0x0/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940754] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940756] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940797] CPU 7 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940798] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940835] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940837] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940840] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940844] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940846] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940847] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940849] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940851] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940852] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940855] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940857] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940858] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940860] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940862] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940864] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940865] Stack: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940866] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940868] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940870] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940873] Call Trace: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940907] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.940953] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941003] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941050] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941096] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941112] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941123] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941135] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941145] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941160] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941177] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941205] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941220] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941235] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941285] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941289] [] ? strlcpy+0x4a/0x60 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941305] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941320] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941335] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941350] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941394] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941434] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941488] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941542] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941596] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941648] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941651] [] ? thread_return+0x4e/0x7d0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941703] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941706] [] ? kthread+0x9e/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941709] [] ? child_rip+0xa/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941712] [] ? kthread+0x0/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941714] [] ? child_rip+0x0/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941715] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941730] Call Trace: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941773] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941819] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941869] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941915] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941962] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941978] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.941988] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942000] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942011] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942026] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942042] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942071] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942085] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942100] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942151] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942154] [] ? strlcpy+0x4a/0x60 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942171] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942185] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942200] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942215] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942259] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942299] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942353] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942407] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942460] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942512] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942516] [] ? thread_return+0x4e/0x7d0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942567] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942571] [] ? kthread+0x9e/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942574] [] ? child_rip+0xa/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942576] [] ? kthread+0x0/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481409.942579] [] ? child_rip+0x0/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.465655] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.562872] CPU 4 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.564985] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.662533] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.664372] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.677925] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.686542] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.692435] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.700392] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.708389] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.716339] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.724330] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.732338] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.741282] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.747669] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.755626] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.763600] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.771539] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.781018] Stack: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.783394] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.791332] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.799931] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.808824] Call Trace: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.811735] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.819209] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.827290] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.834982] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.842471] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.850015] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.857654] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.865038] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.872144] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.879021] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.886868] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.894030] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.902319] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.908679] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.915711] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.923637] [] ? strlcpy+0x4a/0x60 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.929319] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.936181] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.943496] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.950655] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.957938] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.965536] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.973566] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.980993] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.987951] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481412.995787] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.002784] [] ? thread_return+0x4e/0x7d0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.009090] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.016010] [] ? kthread+0x9e/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.021669] [] ? child_rip+0xa/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.027381] [] ? kthread+0x0/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.032927] [] ? child_rip+0x0/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.038645] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.060096] Call Trace: Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.062998] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.070333] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.078424] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.086056] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.093570] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.101131] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.108780] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.116199] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.123254] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.130157] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.138036] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.147912] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.156200] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.162563] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.169661] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.177571] [] ? strlcpy+0x4a/0x60 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.183209] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.190010] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.197358] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.204536] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.211811] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.219426] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.227500] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.234927] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.241921] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.249815] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.256906] [] ? thread_return+0x4e/0x7d0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.263240] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.270161] [] ? kthread+0x9e/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.275785] [] ? child_rip+0xa/0x20 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.281531] [] ? kthread+0x0/0xc0 Aug 25 19:45:51 feral28.ccs.ornl.gov kernel: [1481413.287077] [] ? child_rip+0x0/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.937369] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940369] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940371] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940413] CPU 5 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940414] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940450] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940452] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940455] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940459] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940461] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940463] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940464] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940466] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940468] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940470] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940472] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940473] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940475] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940477] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940479] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940480] Stack: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940481] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940483] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940485] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940487] Call Trace: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940531] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940577] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940627] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940674] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940720] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940736] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940747] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940759] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940770] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940785] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940801] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940830] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940845] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940860] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940910] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940914] [] ? strlcpy+0x4a/0x60 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940930] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940945] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940960] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.940974] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941018] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941059] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941112] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941166] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941220] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941271] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941275] [] ? thread_return+0x4e/0x7d0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941327] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941330] [] ? kthread+0x9e/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941333] [] ? child_rip+0xa/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941336] [] ? kthread+0x0/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941338] [] ? child_rip+0x0/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941339] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941354] Call Trace: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941397] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941443] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941492] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941539] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941586] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941601] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941611] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941623] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941634] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941649] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941665] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941694] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941708] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941723] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941773] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941777] [] ? strlcpy+0x4a/0x60 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941793] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941808] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941823] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941838] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941882] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941922] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.941976] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.942029] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.942083] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.942135] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.942138] [] ? thread_return+0x4e/0x7d0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.942190] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.942193] [] ? kthread+0x9e/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.942196] [] ? child_rip+0xa/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.942199] [] ? kthread+0x0/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.942201] [] ? child_rip+0x0/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944369] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944371] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944414] CPU 6 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944415] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944452] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944453] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944456] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944461] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944462] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944464] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944465] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944467] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944469] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944471] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944473] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944475] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944478] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944480] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944482] Stack: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944483] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944485] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944487] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944489] Call Trace: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944524] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944570] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944620] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944667] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944714] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944729] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944740] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944752] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944764] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944814] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944825] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944840] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944872] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944888] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944917] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944932] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944947] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.944997] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945001] [] ? strlcpy+0x4a/0x60 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945017] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945032] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945047] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945062] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945105] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945146] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945200] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945254] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945307] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945359] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945362] [] ? thread_return+0x4e/0x7d0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945414] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945418] [] ? kthread+0x9e/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945421] [] ? child_rip+0xa/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945423] [] ? kthread+0x0/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945426] [] ? child_rip+0x0/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945427] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945442] Call Trace: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945485] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945531] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945581] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945627] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945674] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945690] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945700] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945712] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945724] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945774] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945785] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945800] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945829] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945845] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945874] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945889] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945904] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945954] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945958] [] ? strlcpy+0x4a/0x60 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945974] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.945989] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946003] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946018] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946062] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946102] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946156] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946211] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946264] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946316] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946319] [] ? thread_return+0x4e/0x7d0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946371] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946375] [] ? kthread+0x9e/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946378] [] ? child_rip+0xa/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946380] [] ? kthread+0x0/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.946382] [] ? child_rip+0x0/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947368] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947370] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947409] CPU 7 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947410] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947446] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947448] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947450] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947454] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947456] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947458] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947459] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947461] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947462] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947464] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947466] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947468] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947469] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947471] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947473] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947474] Stack: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947475] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947477] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947479] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947482] Call Trace: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947516] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947562] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947612] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947659] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947705] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947721] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947732] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947744] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947754] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947769] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947786] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947815] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947829] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947844] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947894] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947898] [] ? strlcpy+0x4a/0x60 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947915] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947929] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947944] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.947959] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948003] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948043] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948097] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948151] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948205] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948256] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948260] [] ? thread_return+0x4e/0x7d0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948312] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948315] [] ? kthread+0x9e/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948318] [] ? child_rip+0xa/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948321] [] ? kthread+0x0/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948323] [] ? child_rip+0x0/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948324] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948339] Call Trace: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948382] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948428] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948478] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948524] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948571] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948586] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948597] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948609] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948619] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948635] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948651] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948679] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948694] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948709] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948759] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948763] [] ? strlcpy+0x4a/0x60 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948779] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948794] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948809] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948823] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948867] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948907] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.948961] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.949015] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.949069] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.949120] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.949124] [] ? thread_return+0x4e/0x7d0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.949175] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.949179] [] ? kthread+0x9e/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.949182] [] ? child_rip+0xa/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.949185] [] ? kthread+0x0/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481493.949187] [] ? child_rip+0x0/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.469143] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.566489] CPU 4 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.568597] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.666160] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.668008] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.681526] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.690238] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.696147] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.704114] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.712069] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.719992] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.727896] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.735801] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.744762] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.751027] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.759028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.766941] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.774811] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.784309] Stack: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.786709] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.794597] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.803090] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.811939] Call Trace: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.814833] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.822263] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.830316] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.837994] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.845534] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.853105] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.860726] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.868086] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.875201] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.882121] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.889939] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.897138] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.908182] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.914597] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.921582] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.929570] [] ? strlcpy+0x4a/0x60 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.935208] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.942026] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.949445] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.956579] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.963871] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.971528] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.979647] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.987056] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481496.994007] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.001857] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.008924] [] ? thread_return+0x4e/0x7d0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.015222] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.022116] [] ? kthread+0x9e/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.027757] [] ? child_rip+0xa/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.033461] [] ? kthread+0x0/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.039014] [] ? child_rip+0x0/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.044752] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.066219] Call Trace: Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.069120] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.076525] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.084641] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.092283] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.099795] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.107324] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.114963] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.122349] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.129463] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.136350] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.144194] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.151401] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.159819] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.166180] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.173159] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.181121] [] ? strlcpy+0x4a/0x60 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.186749] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.193629] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.201012] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.208215] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.215532] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.223122] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.231177] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.238554] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.245566] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.253433] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.260570] [] ? thread_return+0x4e/0x7d0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.266891] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.273779] [] ? kthread+0x9e/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.279403] [] ? child_rip+0xa/0x20 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.285123] [] ? kthread+0x0/0xc0 Aug 25 19:47:15 feral28.ccs.ornl.gov kernel: [1481497.290591] [] ? child_rip+0x0/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.943983] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.946984] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.946985] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947027] CPU 5 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947028] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947064] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947066] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947069] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947073] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947075] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947077] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947078] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947080] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947082] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947084] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947086] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947087] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947091] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947093] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947094] Stack: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947095] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947097] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947099] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947101] Call Trace: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947145] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947191] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947241] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947288] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947334] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947350] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947361] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947373] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947384] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947399] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947415] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947444] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947459] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947474] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947524] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947528] [] ? strlcpy+0x4a/0x60 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947545] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947559] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947574] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947589] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947633] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947673] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947728] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947781] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947835] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947887] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947891] [] ? thread_return+0x4e/0x7d0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947943] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947946] [] ? kthread+0x9e/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947949] [] ? child_rip+0xa/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947952] [] ? kthread+0x0/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947954] [] ? child_rip+0x0/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947955] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.947970] Call Trace: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948013] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948059] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948109] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948156] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948202] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948218] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948228] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948240] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948251] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948266] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948282] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948311] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948325] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948340] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948391] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948394] [] ? strlcpy+0x4a/0x60 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948411] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948425] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948440] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948455] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948499] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948547] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948601] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948655] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948708] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948760] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948764] [] ? thread_return+0x4e/0x7d0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948816] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948819] [] ? kthread+0x9e/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948822] [] ? child_rip+0xa/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948824] [] ? kthread+0x0/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.948826] [] ? child_rip+0x0/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.950984] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.950985] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951029] CPU 6 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951030] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951067] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951069] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951072] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951076] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951078] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951079] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951081] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951083] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951084] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951086] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951088] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951090] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951092] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951094] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951096] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951097] Stack: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951098] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951100] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951102] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951105] Call Trace: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951139] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951186] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951236] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951282] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951329] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951345] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951356] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951368] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951380] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951430] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951441] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951456] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951486] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951502] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951531] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951546] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951561] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951611] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951615] [] ? strlcpy+0x4a/0x60 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951632] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951646] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951661] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951676] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951720] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951760] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951814] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951868] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951922] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951974] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.951977] [] ? thread_return+0x4e/0x7d0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952028] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952032] [] ? kthread+0x9e/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952035] [] ? child_rip+0xa/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952038] [] ? kthread+0x0/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952040] [] ? child_rip+0x0/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952041] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952056] Call Trace: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952099] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952146] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952195] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952242] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952288] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952304] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952315] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952326] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952338] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952388] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952399] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952414] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952443] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952460] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952488] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952503] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952518] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952568] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952572] [] ? strlcpy+0x4a/0x60 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952588] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952603] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952618] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952633] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952676] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952717] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952771] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952824] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952878] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952930] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952933] [] ? thread_return+0x4e/0x7d0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952985] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952988] [] ? kthread+0x9e/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952991] [] ? child_rip+0xa/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952994] [] ? kthread+0x0/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.952996] [] ? child_rip+0x0/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.953983] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.953984] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954023] CPU 7 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954024] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954060] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954061] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954064] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954068] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954070] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954072] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954073] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954075] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954076] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954078] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954080] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954082] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954083] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954085] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954087] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954088] Stack: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954089] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954091] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954093] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954095] Call Trace: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954129] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954176] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954225] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954272] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954319] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954334] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954345] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954357] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954368] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954383] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954399] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954428] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954443] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954457] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954508] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954512] [] ? strlcpy+0x4a/0x60 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954528] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954543] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954557] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954572] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954616] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954656] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954710] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954764] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954817] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954869] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954873] [] ? thread_return+0x4e/0x7d0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954924] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954928] [] ? kthread+0x9e/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954931] [] ? child_rip+0xa/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954934] [] ? kthread+0x0/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954936] [] ? child_rip+0x0/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954937] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954952] Call Trace: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.954995] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955041] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955091] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955137] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955184] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955199] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955210] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955222] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955233] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955248] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955264] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955292] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955307] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955322] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955372] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955376] [] ? strlcpy+0x4a/0x60 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955392] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955407] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955422] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955436] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955480] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955521] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955575] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955629] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955682] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955734] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955737] [] ? thread_return+0x4e/0x7d0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955789] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955792] [] ? kthread+0x9e/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955795] [] ? child_rip+0xa/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955798] [] ? kthread+0x0/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481577.955800] [] ? child_rip+0x0/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.458590] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.555531] CPU 4 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.557635] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.654802] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.656623] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.669953] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.678576] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.684470] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.692355] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.702869] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.710731] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.718590] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.726461] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.735327] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.741611] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.749499] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.757411] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.765262] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.774656] Stack: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.777025] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.784989] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.793535] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.802331] Call Trace: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.805225] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.812578] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.820607] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.828231] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.835710] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.843244] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.850858] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.858184] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.865185] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.872072] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.879847] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.887001] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.895307] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.901626] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.908647] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.916589] [] ? strlcpy+0x4a/0x60 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.922202] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.929074] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.936406] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.943591] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.950900] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.958480] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.966554] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.973999] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.980959] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.988851] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481580.995945] [] ? thread_return+0x4e/0x7d0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.002260] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.009155] [] ? kthread+0x9e/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.014709] [] ? child_rip+0xa/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.020411] [] ? kthread+0x0/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.025915] [] ? child_rip+0x0/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.031590] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.053180] Call Trace: Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.056065] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.063530] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.071559] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.079200] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.086680] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.094188] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.101784] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.109127] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.116138] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.122987] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.130798] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.137986] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.146250] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.152586] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.159554] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.167429] [] ? strlcpy+0x4a/0x60 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.173094] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.179914] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.187272] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.194378] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.201653] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.209233] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.217314] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.224708] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.231625] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.239501] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.246569] [] ? thread_return+0x4e/0x7d0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.252866] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.259803] [] ? kthread+0x9e/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.265391] [] ? child_rip+0xa/0x20 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.271018] [] ? kthread+0x0/0xc0 Aug 25 19:48:39 feral28.ccs.ornl.gov kernel: [1481581.276510] [] ? child_rip+0x0/0x20 Aug 25 19:48:59 feral28.ccs.ornl.gov kernel: [1481601.275121] Lustre: sultan-MDT000b: Client 5d9f30ae-edef-9a3d-c88a-aa425ee6b9cf (at 15@gni1) reconnecting Aug 25 19:48:59 feral28.ccs.ornl.gov kernel: [1481601.285478] Lustre: Skipped 10 previous similar messages Aug 25 19:48:59 feral28.ccs.ornl.gov kernel: [1481601.291384] Lustre: sultan-MDT000b: Connection restored to 62645a1e-aa58-fbd2-2d3b-02416b8b730f (at 15@gni1) Aug 25 19:48:59 feral28.ccs.ornl.gov kernel: [1481601.302022] Lustre: Skipped 14 previous similar messages Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.950598] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953598] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953599] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953642] CPU 5 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953642] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953678] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953680] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953683] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953687] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953689] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953691] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953693] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953694] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953696] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953698] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953700] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953702] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953705] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953707] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953708] Stack: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953709] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953711] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953713] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953715] Call Trace: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953759] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953806] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953856] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953902] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953949] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953965] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953976] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953988] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.953998] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954013] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954029] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954059] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954073] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954088] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954139] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954143] [] ? strlcpy+0x4a/0x60 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954159] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954174] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954189] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954204] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954247] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954288] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954342] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954395] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954449] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954500] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954504] [] ? thread_return+0x4e/0x7d0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954556] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954559] [] ? kthread+0x9e/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954562] [] ? child_rip+0xa/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954565] [] ? kthread+0x0/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954567] [] ? child_rip+0x0/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954568] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954583] Call Trace: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954626] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954672] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954722] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954768] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954815] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954831] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954841] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954853] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954863] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954878] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954895] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954923] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954938] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.954953] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955003] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955007] [] ? strlcpy+0x4a/0x60 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955023] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955037] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955052] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955067] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955111] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955152] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955206] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955260] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955313] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955365] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955368] [] ? thread_return+0x4e/0x7d0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955420] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955423] [] ? kthread+0x9e/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955426] [] ? child_rip+0xa/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955429] [] ? kthread+0x0/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.955431] [] ? child_rip+0x0/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957598] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957599] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957642] CPU 6 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957643] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957681] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957683] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957686] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957690] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957692] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957693] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957695] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957697] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957699] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957701] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957703] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957704] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957706] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957708] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957710] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957711] Stack: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957712] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957714] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957716] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957719] Call Trace: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957754] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957800] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957850] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957897] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957944] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957959] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957970] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957981] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.957994] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958044] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958055] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958070] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958101] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958118] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958147] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958161] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958176] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958227] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958231] [] ? strlcpy+0x4a/0x60 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958247] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958262] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958277] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958292] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958335] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958376] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958430] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958483] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958537] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958589] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958593] [] ? thread_return+0x4e/0x7d0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958645] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958648] [] ? kthread+0x9e/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958651] [] ? child_rip+0xa/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958654] [] ? kthread+0x0/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958656] [] ? child_rip+0x0/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958657] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958672] Call Trace: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958715] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958761] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958811] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958858] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958904] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958920] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958930] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958942] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.958954] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959004] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959015] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959031] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959061] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959078] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959106] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959121] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959136] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959186] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959190] [] ? strlcpy+0x4a/0x60 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959206] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959221] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959235] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959250] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959294] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959334] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959388] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959442] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959496] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959548] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959551] [] ? thread_return+0x4e/0x7d0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959603] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959607] [] ? kthread+0x9e/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959610] [] ? child_rip+0xa/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959613] [] ? kthread+0x0/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.959615] [] ? child_rip+0x0/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960597] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960598] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960638] CPU 7 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960639] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960674] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960676] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960679] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960683] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960685] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960686] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960688] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960689] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960691] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960693] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960695] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960696] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960700] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960701] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960703] Stack: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960704] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960706] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960708] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960711] Call Trace: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960745] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960791] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960841] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960888] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960935] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960950] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960961] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960973] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960984] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.960999] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961015] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961044] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961059] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961074] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961124] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961128] [] ? strlcpy+0x4a/0x60 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961144] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961159] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961174] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961189] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961232] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961273] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961327] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961380] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961434] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961486] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961489] [] ? thread_return+0x4e/0x7d0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961541] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961545] [] ? kthread+0x9e/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961548] [] ? child_rip+0xa/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961550] [] ? kthread+0x0/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961553] [] ? child_rip+0x0/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961554] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961569] Call Trace: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961612] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961658] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961708] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961754] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961801] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961816] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961827] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961839] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961849] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961865] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961881] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961909] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961924] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961939] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961989] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.961993] [] ? strlcpy+0x4a/0x60 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962009] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962024] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962039] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962054] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962097] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962138] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962192] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962246] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962300] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962352] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962355] [] ? thread_return+0x4e/0x7d0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962407] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962410] [] ? kthread+0x9e/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962413] [] ? child_rip+0xa/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962416] [] ? kthread+0x0/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481661.962418] [] ? child_rip+0x0/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.486815] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.584007] CPU 4 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.586113] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.683536] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.685418] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.698965] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.707744] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.713664] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.721543] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.729481] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.737352] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.745282] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.753229] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.762156] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.768535] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.776465] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.784439] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.792326] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.801833] Stack: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.804277] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.812207] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.820753] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.829566] Call Trace: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.832470] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.839917] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.848024] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.855743] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.863232] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.870681] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.878338] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.885706] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.892863] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.899784] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.907663] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.914852] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.923157] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.929460] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.936445] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.944364] [] ? strlcpy+0x4a/0x60 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.949976] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.956787] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.964102] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.971312] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.978639] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.986210] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481664.994276] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.001668] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.008628] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.016487] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.023502] [] ? thread_return+0x4e/0x7d0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.029834] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.036702] [] ? kthread+0x9e/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.042290] [] ? child_rip+0xa/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.047986] [] ? kthread+0x0/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.053524] [] ? child_rip+0x0/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.059225] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.080728] Call Trace: Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.083630] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.091000] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.099055] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.106713] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.114269] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.121789] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.129480] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.136813] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.143910] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.150841] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.158710] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.165934] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.174206] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.180594] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.187607] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.195594] [] ? strlcpy+0x4a/0x60 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.201247] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.208058] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.215417] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.222525] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.229860] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.237492] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.245644] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.253054] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.259970] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.267829] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.274905] [] ? thread_return+0x4e/0x7d0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.281228] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.288131] [] ? kthread+0x9e/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.293764] [] ? child_rip+0xa/0x20 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.299508] [] ? kthread+0x0/0xc0 Aug 25 19:50:03 feral28.ccs.ornl.gov kernel: [1481665.305022] [] ? child_rip+0x0/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.957212] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960212] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960214] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960256] CPU 5 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960257] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960294] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960295] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960298] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960302] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960304] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960306] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960307] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960309] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960311] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960313] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960315] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960316] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960318] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960320] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960322] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960323] Stack: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960324] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960326] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960328] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960330] Call Trace: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960374] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960420] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960470] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960517] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960564] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960579] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960590] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960602] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960613] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960628] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960644] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960674] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960688] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960703] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960754] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960758] [] ? strlcpy+0x4a/0x60 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960774] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960788] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960803] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960818] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960862] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960902] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.960957] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961011] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961064] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961116] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961120] [] ? thread_return+0x4e/0x7d0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961171] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961175] [] ? kthread+0x9e/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961178] [] ? child_rip+0xa/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961181] [] ? kthread+0x0/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961183] [] ? child_rip+0x0/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961184] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961199] Call Trace: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961242] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961288] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961338] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961384] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961431] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961447] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961457] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961469] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961479] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961494] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961510] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961539] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961554] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961569] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961619] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961623] [] ? strlcpy+0x4a/0x60 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961639] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961654] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961669] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961683] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961727] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961768] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961822] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961876] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961929] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961981] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.961985] [] ? thread_return+0x4e/0x7d0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.962037] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.962040] [] ? kthread+0x9e/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.962043] [] ? child_rip+0xa/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.962046] [] ? kthread+0x0/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.962048] [] ? child_rip+0x0/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964212] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964214] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964256] CPU 6 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964257] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964294] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964296] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964299] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964304] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964306] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964307] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964309] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964311] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964312] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964315] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964317] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964318] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964320] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964322] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964324] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964325] Stack: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964326] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964328] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964330] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964332] Call Trace: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964367] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964414] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964464] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964510] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964557] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964573] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964583] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964595] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964607] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964657] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964668] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964684] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964714] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964731] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964759] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964774] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964789] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964839] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964843] [] ? strlcpy+0x4a/0x60 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964860] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964874] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964889] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964904] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964948] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.964988] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965043] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965096] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965150] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965202] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965205] [] ? thread_return+0x4e/0x7d0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965257] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965260] [] ? kthread+0x9e/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965263] [] ? child_rip+0xa/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965266] [] ? kthread+0x0/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965268] [] ? child_rip+0x0/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965269] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965285] Call Trace: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965327] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965374] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965423] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965470] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965516] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965532] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965542] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965554] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965566] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965616] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965627] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965642] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965671] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965688] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965716] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965731] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965746] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965796] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965800] [] ? strlcpy+0x4a/0x60 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965816] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965831] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965846] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965860] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965904] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965945] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.965999] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.966053] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.966106] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.966158] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.966162] [] ? thread_return+0x4e/0x7d0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.966214] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.966217] [] ? kthread+0x9e/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.966220] [] ? child_rip+0xa/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.966223] [] ? kthread+0x0/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.966225] [] ? child_rip+0x0/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967211] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967213] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967253] CPU 7 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967254] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967290] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967291] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967294] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967299] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967300] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967302] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967303] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967305] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967306] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967308] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967310] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967312] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967313] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967315] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967317] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967318] Stack: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967319] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967321] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967324] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967326] Call Trace: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967360] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967407] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967456] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967503] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967549] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967565] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967575] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967587] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967598] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967613] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967629] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967658] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967673] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967688] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967738] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967742] [] ? strlcpy+0x4a/0x60 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967758] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967773] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967788] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967803] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967847] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967887] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967941] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.967995] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968049] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968101] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968104] [] ? thread_return+0x4e/0x7d0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968156] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968159] [] ? kthread+0x9e/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968162] [] ? child_rip+0xa/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968165] [] ? kthread+0x0/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968167] [] ? child_rip+0x0/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968168] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968184] Call Trace: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968226] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968272] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968322] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968369] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968415] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968431] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968441] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968453] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968463] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968479] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968495] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968523] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968538] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968553] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968603] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968607] [] ? strlcpy+0x4a/0x60 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968623] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968638] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968653] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968667] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968711] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968752] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968806] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968860] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968913] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968965] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.968969] [] ? thread_return+0x4e/0x7d0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.969021] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.969024] [] ? kthread+0x9e/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.969027] [] ? child_rip+0xa/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.969029] [] ? kthread+0x0/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481745.969032] [] ? child_rip+0x0/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.493285] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.590640] CPU 4 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.592738] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.690276] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.692149] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.705600] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.714284] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.720229] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.728151] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.736090] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.744003] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.751897] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.759847] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.768782] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.775099] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.782996] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.790961] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.798822] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.808347] Stack: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.810773] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.818729] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.827214] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.836176] Call Trace: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.839164] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.846568] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.854605] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.862186] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.869598] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.877116] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.884685] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.891991] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.899020] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.905853] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.913629] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.920803] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.929082] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.935410] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.942352] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.950243] [] ? strlcpy+0x4a/0x60 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.955873] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.962710] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.970033] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.977141] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.984519] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481748.992097] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.000208] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.007643] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.014655] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.022532] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.029537] [] ? thread_return+0x4e/0x7d0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.035913] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.042877] [] ? kthread+0x9e/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.048450] [] ? child_rip+0xa/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.054091] [] ? kthread+0x0/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.059586] [] ? child_rip+0x0/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.065278] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.086894] Call Trace: Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.089787] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.097218] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.105230] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.112871] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.120290] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.127773] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.135361] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.142720] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.149773] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.156669] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.164495] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.171677] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.179912] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.186274] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.193314] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.201241] [] ? strlcpy+0x4a/0x60 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.206861] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.213671] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.221038] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.228282] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.235575] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.243225] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.253977] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.261378] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.268371] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.276223] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.283323] [] ? thread_return+0x4e/0x7d0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.289640] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.296479] [] ? kthread+0x9e/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.302069] [] ? child_rip+0xa/0x20 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.307782] [] ? kthread+0x0/0xc0 Aug 25 19:51:27 feral28.ccs.ornl.gov kernel: [1481749.313285] [] ? child_rip+0x0/0x20 Aug 25 19:51:55 feral28.ccs.ornl.gov kernel: [1481777.216672] Lustre: sultan-MDT000b: haven't heard from client 121a88f7-96bf-42c6-fa63-63b545c0ecc1 (at 7@gni1) in 312 seconds. I think it's dead, and I am evicting it. exp ffff880bade16c00, cur 1472169115 expire 1472168893 last 1472168803 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.963827] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966827] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966828] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966870] CPU 5 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966871] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966908] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966910] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966913] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966917] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966919] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966921] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966922] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966924] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966926] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966928] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966930] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966931] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966933] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966935] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966937] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966938] Stack: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966939] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966941] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966943] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966946] Call Trace: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.966990] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967036] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967086] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967132] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967179] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967195] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967206] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967218] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967228] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967243] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967259] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967289] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967304] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967319] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967369] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967373] [] ? strlcpy+0x4a/0x60 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967389] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967404] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967419] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967433] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967477] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967517] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967571] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967625] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967679] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967731] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967734] [] ? thread_return+0x4e/0x7d0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967786] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967789] [] ? kthread+0x9e/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967792] [] ? child_rip+0xa/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967795] [] ? kthread+0x0/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967797] [] ? child_rip+0x0/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967798] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967814] Call Trace: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967856] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967903] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967952] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.967999] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968045] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968061] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968071] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968083] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968094] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968109] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968125] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968153] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968168] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968183] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968233] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968237] [] ? strlcpy+0x4a/0x60 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968253] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968268] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968283] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968298] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968341] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968382] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968436] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968490] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968543] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968595] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968598] [] ? thread_return+0x4e/0x7d0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968650] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968654] [] ? kthread+0x9e/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968656] [] ? child_rip+0xa/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968659] [] ? kthread+0x0/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.968661] [] ? child_rip+0x0/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970827] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970828] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970871] CPU 6 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970872] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970909] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970911] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970914] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970919] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970921] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970922] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970924] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970926] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970927] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970930] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970932] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970933] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970937] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970939] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970940] Stack: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970941] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970943] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970945] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970948] Call Trace: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.970983] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971029] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971079] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971126] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971173] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971188] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971199] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971211] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971223] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971273] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971284] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971299] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971330] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971347] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971376] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971390] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971405] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971455] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971459] [] ? strlcpy+0x4a/0x60 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971476] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971490] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971505] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971520] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971564] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971612] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971666] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971721] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971774] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971826] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971830] [] ? thread_return+0x4e/0x7d0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971881] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971885] [] ? kthread+0x9e/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971888] [] ? child_rip+0xa/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971890] [] ? kthread+0x0/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971892] [] ? child_rip+0x0/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971894] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971909] Call Trace: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971952] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.971998] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972048] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972094] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972141] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972157] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972167] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972179] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972191] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972241] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972252] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972267] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972296] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972313] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972341] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972356] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972371] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972421] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972425] [] ? strlcpy+0x4a/0x60 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972441] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972456] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972471] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972485] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972529] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972569] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972623] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972677] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972730] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972782] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972786] [] ? thread_return+0x4e/0x7d0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972838] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972841] [] ? kthread+0x9e/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972844] [] ? child_rip+0xa/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972847] [] ? kthread+0x0/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.972849] [] ? child_rip+0x0/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973826] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973827] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973868] CPU 7 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973868] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973904] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973906] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973909] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973913] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973915] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973916] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973918] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973919] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973921] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973923] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973925] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973926] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973928] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973930] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973932] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973933] Stack: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973934] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973936] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973938] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973941] Call Trace: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.973975] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974021] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974071] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974117] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974164] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974179] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974190] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974202] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974213] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974228] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974244] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974273] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974288] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974303] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974353] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974357] [] ? strlcpy+0x4a/0x60 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974373] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974388] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974403] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974417] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974461] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974502] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974556] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974609] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974663] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974715] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974718] [] ? thread_return+0x4e/0x7d0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974770] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974773] [] ? kthread+0x9e/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974776] [] ? child_rip+0xa/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974779] [] ? kthread+0x0/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974781] [] ? child_rip+0x0/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974782] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974797] Call Trace: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974840] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974886] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974936] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.974983] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975029] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975045] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975055] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975067] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975078] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975093] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975109] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975137] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975152] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975167] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975217] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975221] [] ? strlcpy+0x4a/0x60 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975237] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975252] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975266] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975281] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975325] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975365] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975419] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975473] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975527] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975578] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975582] [] ? thread_return+0x4e/0x7d0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975634] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975637] [] ? kthread+0x9e/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975640] [] ? child_rip+0xa/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975643] [] ? kthread+0x0/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481829.975645] [] ? child_rip+0x0/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.497490] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.594753] CPU 4 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.596894] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.694232] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.696105] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.709555] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.718267] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.724071] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.732021] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.739977] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.747890] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.755829] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.763777] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.772773] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.779142] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.787307] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.795341] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.803272] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.812753] Stack: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.815197] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.823111] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.831639] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.840487] Call Trace: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.843390] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.850856] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.858953] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.866621] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.874108] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.881696] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.889300] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.896678] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.903792] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.910720] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.918601] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.925747] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.934009] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.940378] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.947470] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.955431] [] ? strlcpy+0x4a/0x60 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.961016] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.967835] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.975221] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.982397] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.989715] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481832.999998] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.008004] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.015423] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.022452] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.030423] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.037499] [] ? thread_return+0x4e/0x7d0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.043840] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.050775] [] ? kthread+0x9e/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.056427] [] ? child_rip+0xa/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.062174] [] ? kthread+0x0/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.067720] [] ? child_rip+0x0/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.073456] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.094881] Call Trace: Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.097816] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.105196] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.113251] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.120987] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.128475] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.136028] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.143702] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.151027] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.158167] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.165061] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.172924] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.180148] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.188420] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.194806] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.201802] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.209695] [] ? strlcpy+0x4a/0x60 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.215393] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.222256] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.229683] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.236885] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.244246] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.251853] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.259935] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.267363] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.274296] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.282198] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.289326] [] ? thread_return+0x4e/0x7d0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.295650] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.302526] [] ? kthread+0x9e/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.308133] [] ? child_rip+0xa/0x20 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.313845] [] ? kthread+0x0/0xc0 Aug 25 19:52:51 feral28.ccs.ornl.gov kernel: [1481833.319365] [] ? child_rip+0x0/0x20 Aug 25 19:54:03 feral28.ccs.ornl.gov kernel: [1481905.984981] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 19:54:03 feral28.ccs.ornl.gov kernel: [1481905.996028] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 37 previous similar messages Aug 25 19:54:03 feral28.ccs.ornl.gov kernel: [1481906.007052] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.119@o2ib1 (222): c: 63, oc: 0, rc: 63 Aug 25 19:54:03 feral28.ccs.ornl.gov kernel: [1481906.020495] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 37 previous similar messages Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.970441] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973441] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973442] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973485] CPU 5 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973486] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973521] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973523] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973526] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973530] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973532] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973534] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973535] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973537] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973539] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973541] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973543] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973544] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973548] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973550] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973551] Stack: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973552] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973554] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973556] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973559] Call Trace: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973602] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973649] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973698] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973745] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973792] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973807] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973818] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973830] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973841] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973856] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973872] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973902] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973916] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973931] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973982] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.973986] [] ? strlcpy+0x4a/0x60 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974002] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974017] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974032] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974047] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974091] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974131] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974185] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974239] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974293] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974345] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974348] [] ? thread_return+0x4e/0x7d0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974400] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974404] [] ? kthread+0x9e/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974406] [] ? child_rip+0xa/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974409] [] ? kthread+0x0/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974411] [] ? child_rip+0x0/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974413] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974428] Call Trace: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974471] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974517] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974567] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974613] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974660] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974675] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974686] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974698] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974708] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974724] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974740] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974768] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974783] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974798] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974848] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974852] [] ? strlcpy+0x4a/0x60 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974868] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974883] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974898] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974912] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974956] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.974997] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975051] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975105] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975159] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975211] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975214] [] ? thread_return+0x4e/0x7d0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975266] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975269] [] ? kthread+0x9e/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975272] [] ? child_rip+0xa/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975275] [] ? kthread+0x0/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.975277] [] ? child_rip+0x0/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977441] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977443] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977485] CPU 6 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977486] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977523] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977525] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977528] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977533] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977534] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977536] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977538] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977539] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977541] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977543] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977545] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977547] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977550] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977552] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977554] Stack: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977555] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977557] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977559] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977561] Call Trace: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977596] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977642] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977692] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977739] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977786] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977801] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977812] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977824] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977836] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977886] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977897] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977912] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977943] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977960] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.977989] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978004] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978018] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978069] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978073] [] ? strlcpy+0x4a/0x60 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978089] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978104] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978119] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978134] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978178] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978218] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978272] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978326] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978380] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978432] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978435] [] ? thread_return+0x4e/0x7d0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978486] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978490] [] ? kthread+0x9e/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978493] [] ? child_rip+0xa/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978496] [] ? kthread+0x0/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978498] [] ? child_rip+0x0/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978499] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978514] Call Trace: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978557] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978603] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978653] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978700] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978746] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978762] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978773] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978784] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978796] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978846] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978857] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978872] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978902] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978918] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978947] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978961] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.978976] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979027] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979030] [] ? strlcpy+0x4a/0x60 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979047] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979062] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979076] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979091] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979135] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979175] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979229] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979283] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979337] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979388] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979392] [] ? thread_return+0x4e/0x7d0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979444] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979447] [] ? kthread+0x9e/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979450] [] ? child_rip+0xa/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979453] [] ? kthread+0x0/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.979455] [] ? child_rip+0x0/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980440] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980441] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980482] CPU 7 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980483] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980519] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980520] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980523] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980527] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980529] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980531] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980532] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980534] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980535] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980537] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980539] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980541] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980544] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980546] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980547] Stack: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980548] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980550] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980553] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980555] Call Trace: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980589] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980636] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980685] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980732] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980779] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980794] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980805] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980817] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980828] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980843] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980859] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980888] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980903] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980918] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980968] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980972] [] ? strlcpy+0x4a/0x60 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.980988] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981003] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981018] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981032] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981076] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981117] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981170] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981224] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981278] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981330] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981333] [] ? thread_return+0x4e/0x7d0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981385] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981388] [] ? kthread+0x9e/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981391] [] ? child_rip+0xa/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981394] [] ? kthread+0x0/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981396] [] ? child_rip+0x0/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981397] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981413] Call Trace: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981455] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981502] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981551] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981598] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981644] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981660] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981671] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981683] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981693] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981708] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981724] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981753] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981768] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981783] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981833] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981837] [] ? strlcpy+0x4a/0x60 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981853] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981868] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981882] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981897] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981941] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.981981] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982035] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982089] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982143] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982195] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982198] [] ? thread_return+0x4e/0x7d0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982250] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982254] [] ? kthread+0x9e/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982256] [] ? child_rip+0xa/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982259] [] ? kthread+0x0/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481913.982261] [] ? child_rip+0x0/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.504456] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.601778] CPU 4 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.603874] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.701706] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.703546] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.716970] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.725680] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.731538] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.739513] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.747426] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.758000] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.765957] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.773895] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.782866] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.789227] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.797132] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.805087] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.813035] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.822585] Stack: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.824986] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.832917] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.841437] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.850301] Call Trace: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.853196] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.860679] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.868751] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.876497] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.884035] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.891615] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.899281] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.906719] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.913814] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.920762] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.928580] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.935770] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.944144] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.950559] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.957632] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.965542] [] ? strlcpy+0x4a/0x60 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.971213] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.977999] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.985408] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.992619] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481916.999944] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.007500] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.015609] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.022984] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.029883] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.037802] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.044913] [] ? thread_return+0x4e/0x7d0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.051176] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.058096] [] ? kthread+0x9e/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.063703] [] ? child_rip+0xa/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.069457] [] ? kthread+0x0/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.075004] [] ? child_rip+0x0/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.080688] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.102165] Call Trace: Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.105049] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.112463] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.120588] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.128315] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.135804] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.143391] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.151048] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.158399] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.165478] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.172434] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.180313] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.187474] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.195756] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.202109] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.209147] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.217091] [] ? strlcpy+0x4a/0x60 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.222747] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.229584] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.236933] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.244204] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.251573] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.259188] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.267281] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.274638] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.281650] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.289543] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.296697] [] ? thread_return+0x4e/0x7d0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.302986] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.309871] [] ? kthread+0x9e/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.315521] [] ? child_rip+0xa/0x20 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.321207] [] ? kthread+0x0/0xc0 Aug 25 19:54:15 feral28.ccs.ornl.gov kernel: [1481917.326727] [] ? child_rip+0x0/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.977055] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980055] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980057] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980099] CPU 5 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980100] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980136] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980137] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980140] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980145] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980146] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980148] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980149] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980151] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980152] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980154] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980156] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980158] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980160] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980162] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980164] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980165] Stack: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980166] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980168] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980170] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980173] Call Trace: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980216] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980263] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980313] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980359] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980406] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980422] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980433] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980445] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980455] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980471] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980487] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980516] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980530] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980545] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980596] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980600] [] ? strlcpy+0x4a/0x60 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980616] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980631] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980646] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980661] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980704] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980745] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980799] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980853] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980906] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980958] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.980962] [] ? thread_return+0x4e/0x7d0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981013] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981017] [] ? kthread+0x9e/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981020] [] ? child_rip+0xa/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981022] [] ? kthread+0x0/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981025] [] ? child_rip+0x0/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981026] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981041] Call Trace: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981084] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981130] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981180] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981226] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981273] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981289] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981299] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981311] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981322] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981337] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981353] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981382] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981396] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981411] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981462] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981466] [] ? strlcpy+0x4a/0x60 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981482] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981496] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981511] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981526] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981570] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981610] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981664] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981718] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981772] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981824] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981827] [] ? thread_return+0x4e/0x7d0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981879] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981882] [] ? kthread+0x9e/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981885] [] ? child_rip+0xa/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981888] [] ? kthread+0x0/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.981890] [] ? child_rip+0x0/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984055] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984057] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984100] CPU 6 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984101] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984137] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984139] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984142] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984146] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984148] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984150] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984151] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984153] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984155] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984157] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984159] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984161] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984164] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984166] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984167] Stack: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984168] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984170] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984172] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984175] Call Trace: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984210] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984256] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984306] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984353] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984399] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984415] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984426] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984438] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984450] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984500] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984511] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984526] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984555] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984572] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984601] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984615] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984630] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984681] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984685] [] ? strlcpy+0x4a/0x60 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984701] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984716] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984730] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984745] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984789] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984829] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984883] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984937] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.984991] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985043] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985046] [] ? thread_return+0x4e/0x7d0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985098] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985102] [] ? kthread+0x9e/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985105] [] ? child_rip+0xa/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985107] [] ? kthread+0x0/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985109] [] ? child_rip+0x0/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985111] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985126] Call Trace: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985169] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985215] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985265] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985311] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985358] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985373] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985384] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985396] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985408] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985458] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985468] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985484] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985513] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985529] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985558] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985573] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985587] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985638] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985642] [] ? strlcpy+0x4a/0x60 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985658] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985673] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985688] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985703] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985746] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985787] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985841] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985894] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985948] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.985999] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.986003] [] ? thread_return+0x4e/0x7d0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.986054] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.986058] [] ? kthread+0x9e/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.986061] [] ? child_rip+0xa/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.986064] [] ? kthread+0x0/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.986066] [] ? child_rip+0x0/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987054] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987056] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987095] CPU 7 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987096] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987133] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987134] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987137] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987141] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987143] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987144] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987146] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987147] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987149] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987151] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987153] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987154] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987156] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987158] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987160] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987161] Stack: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987162] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987164] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987166] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987168] Call Trace: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987202] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987248] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987298] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987345] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987391] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987407] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987418] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987430] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987440] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987455] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987471] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987500] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987515] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987530] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987580] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987584] [] ? strlcpy+0x4a/0x60 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987601] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987615] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987630] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987645] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987689] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987729] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987783] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987837] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987890] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987942] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987946] [] ? thread_return+0x4e/0x7d0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.987997] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988001] [] ? kthread+0x9e/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988004] [] ? child_rip+0xa/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988007] [] ? kthread+0x0/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988009] [] ? child_rip+0x0/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988010] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988025] Call Trace: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988068] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988114] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988164] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988210] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988257] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988272] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988283] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988295] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988305] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988320] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988336] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988365] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988379] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988394] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988444] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988448] [] ? strlcpy+0x4a/0x60 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988464] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988479] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988494] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988509] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988553] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988593] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988647] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988701] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988755] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988807] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988810] [] ? thread_return+0x4e/0x7d0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988862] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988865] [] ? kthread+0x9e/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988868] [] ? child_rip+0xa/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988871] [] ? kthread+0x0/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1481997.988873] [] ? child_rip+0x0/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.512399] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.609634] CPU 4 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.611740] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.709380] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.711246] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.724723] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.733457] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.739430] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.747396] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.755307] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.763169] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.771115] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.779117] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.788087] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.794449] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.802396] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.810361] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.818256] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.827764] Stack: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.830148] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.838069] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.846580] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.855429] Call Trace: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.858332] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.865702] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.873842] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.881562] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.889085] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.896654] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.904320] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.911688] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.918784] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.925742] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.933551] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.940801] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.949098] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.955493] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.962533] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.970504] [] ? strlcpy+0x4a/0x60 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.976184] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.983012] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.990414] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482000.997614] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.004889] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.012522] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.020623] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.028075] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.035026] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.042981] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.050126] [] ? thread_return+0x4e/0x7d0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.056467] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.063369] [] ? kthread+0x9e/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.069029] [] ? child_rip+0xa/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.074714] [] ? kthread+0x0/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.080269] [] ? child_rip+0x0/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.086031] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.107448] Call Trace: Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.110349] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.117762] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.125836] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.133459] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.140912] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.148499] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.156173] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.163533] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.170638] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.177499] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.185351] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.192551] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.200865] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.207191] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.214255] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.222175] [] ? strlcpy+0x4a/0x60 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.227794] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.234527] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.241955] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.249209] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.256588] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.264227] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.272268] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.279711] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.286662] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.294557] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.301659] [] ? thread_return+0x4e/0x7d0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.308060] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.314954] [] ? kthread+0x9e/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.320544] [] ? child_rip+0xa/0x20 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.326256] [] ? kthread+0x0/0xc0 Aug 25 19:55:39 feral28.ccs.ornl.gov kernel: [1482001.331828] [] ? child_rip+0x0/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.983670] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986670] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986671] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986713] CPU 5 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986714] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986750] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986752] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986754] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986759] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986760] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986762] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986763] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986765] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986767] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986768] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986770] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986772] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986775] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986777] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986779] Stack: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986780] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986782] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986784] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986786] Call Trace: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986830] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986877] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986926] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.986973] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987020] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987036] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987046] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987058] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987069] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987084] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987100] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987129] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987144] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987159] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987209] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987213] [] ? strlcpy+0x4a/0x60 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987230] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987245] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987259] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987274] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987318] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987359] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987413] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987467] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987521] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987573] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987576] [] ? thread_return+0x4e/0x7d0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987628] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987631] [] ? kthread+0x9e/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987634] [] ? child_rip+0xa/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987637] [] ? kthread+0x0/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987639] [] ? child_rip+0x0/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987640] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987656] Call Trace: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987698] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987745] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987794] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987841] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987887] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987903] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987913] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987925] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987936] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987951] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987967] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.987996] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988010] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988025] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988075] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988079] [] ? strlcpy+0x4a/0x60 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988095] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988110] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988125] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988140] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988184] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988224] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988278] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988332] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988386] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988438] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988441] [] ? thread_return+0x4e/0x7d0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988493] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988496] [] ? kthread+0x9e/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988499] [] ? child_rip+0xa/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988502] [] ? kthread+0x0/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.988504] [] ? child_rip+0x0/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990670] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990671] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990714] CPU 6 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990715] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990752] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990754] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990757] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990762] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990763] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990765] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990766] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990768] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990770] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990772] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990774] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990776] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990777] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990779] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990781] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990782] Stack: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990783] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990785] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990787] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990790] Call Trace: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990825] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990871] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990921] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.990968] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991015] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991030] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991041] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991053] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991065] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991115] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991126] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991142] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991173] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991189] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991218] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991233] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991248] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991299] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991303] [] ? strlcpy+0x4a/0x60 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991319] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991333] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991348] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991363] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991407] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991447] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991502] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991555] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991609] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991661] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991665] [] ? thread_return+0x4e/0x7d0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991717] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991720] [] ? kthread+0x9e/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991723] [] ? child_rip+0xa/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991726] [] ? kthread+0x0/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991728] [] ? child_rip+0x0/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991729] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991745] Call Trace: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991787] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991834] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991883] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991930] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991976] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.991992] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992002] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992014] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992026] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992077] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992087] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992102] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992131] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992148] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992176] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992191] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992206] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992257] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992260] [] ? strlcpy+0x4a/0x60 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992277] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992292] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992306] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992321] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992365] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992405] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992459] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992513] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992567] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992619] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992622] [] ? thread_return+0x4e/0x7d0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992674] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992677] [] ? kthread+0x9e/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992680] [] ? child_rip+0xa/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992683] [] ? kthread+0x0/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.992685] [] ? child_rip+0x0/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993669] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993670] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993711] CPU 7 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993712] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993749] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993751] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993754] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993758] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993759] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993761] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993763] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993764] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993766] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993768] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993770] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993772] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993775] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993777] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993779] Stack: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993779] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993781] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993784] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993786] Call Trace: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993820] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993866] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993916] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.993963] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994009] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994025] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994036] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994048] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994058] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994073] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994089] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994119] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994133] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994148] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994199] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994203] [] ? strlcpy+0x4a/0x60 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994219] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994234] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994249] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994263] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994307] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994348] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994402] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994456] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994509] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994561] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994565] [] ? thread_return+0x4e/0x7d0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994617] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994620] [] ? kthread+0x9e/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994623] [] ? child_rip+0xa/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994625] [] ? kthread+0x0/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994628] [] ? child_rip+0x0/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994629] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994644] Call Trace: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994686] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994733] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994782] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994829] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994876] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994891] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994902] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994914] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994924] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994939] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994955] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994984] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.994999] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995014] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995064] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995067] [] ? strlcpy+0x4a/0x60 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995084] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995098] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995113] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995128] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995172] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995212] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995266] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995320] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995374] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995425] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995429] [] ? thread_return+0x4e/0x7d0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995481] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995484] [] ? kthread+0x9e/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995487] [] ? child_rip+0xa/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995489] [] ? kthread+0x0/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482081.995491] [] ? child_rip+0x0/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.520697] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.617889] CPU 4 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.619976] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.717498] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.719388] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.732916] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.741637] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.747582] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.755503] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.763433] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.771408] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.779294] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.787217] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.796212] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.802591] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.810548] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.818495] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.826418] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.835958] Stack: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.838316] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.846290] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.854836] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.863710] Call Trace: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.866621] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.874001] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.882045] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.889696] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.897227] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.904754] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.912411] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.919822] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.926935] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.933849] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.941711] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.948865] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.957162] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.963507] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.970624] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.978559] [] ? strlcpy+0x4a/0x60 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.984169] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.991041] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482084.998461] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.005619] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.012947] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.020569] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.028713] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.036088] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.043091] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.051019] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.058113] [] ? thread_return+0x4e/0x7d0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.064480] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.071321] [] ? kthread+0x9e/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.076971] [] ? child_rip+0xa/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.082683] [] ? kthread+0x0/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.088195] [] ? child_rip+0x0/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.093861] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.115287] Call Trace: Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.118224] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.125680] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.133752] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.141478] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.148985] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.156563] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.164178] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.171546] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.178626] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.185572] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.193409] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.200588] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.208956] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.215352] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.222407] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.230361] [] ? strlcpy+0x4a/0x60 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.236006] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.242800] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.250124] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.257316] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.264627] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.272216] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.280238] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.287620] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.294589] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.302474] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.309567] [] ? thread_return+0x4e/0x7d0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.315924] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.322845] [] ? kthread+0x9e/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.331122] [] ? child_rip+0xa/0x20 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.336834] [] ? kthread+0x0/0xc0 Aug 25 19:57:03 feral28.ccs.ornl.gov kernel: [1482085.342371] [] ? child_rip+0x0/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.990284] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993284] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993286] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993329] CPU 5 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993329] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993365] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993367] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993370] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993374] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993376] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993377] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993379] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993380] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993382] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993384] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993386] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993387] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993389] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993391] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993393] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993394] Stack: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993395] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993397] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993399] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993402] Call Trace: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993446] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993493] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993542] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993589] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993636] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993651] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993662] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993674] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993685] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993700] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993716] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993745] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993760] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993775] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993825] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993829] [] ? strlcpy+0x4a/0x60 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993846] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993860] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993875] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993890] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993934] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.993974] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994029] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994082] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994136] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994188] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994191] [] ? thread_return+0x4e/0x7d0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994243] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994247] [] ? kthread+0x9e/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994249] [] ? child_rip+0xa/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994252] [] ? kthread+0x0/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994254] [] ? child_rip+0x0/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994256] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994271] Call Trace: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994313] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994360] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994410] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994456] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994503] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994518] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994529] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994541] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994551] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994566] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994582] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994611] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994626] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994641] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994691] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994695] [] ? strlcpy+0x4a/0x60 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994711] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994726] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994741] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994756] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994799] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994840] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994894] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.994948] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.995001] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.995053] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.995057] [] ? thread_return+0x4e/0x7d0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.995108] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.995112] [] ? kthread+0x9e/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.995115] [] ? child_rip+0xa/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.995117] [] ? kthread+0x0/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.995120] [] ? child_rip+0x0/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997284] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997286] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997329] CPU 6 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997329] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997365] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997367] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997370] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997375] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997376] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997378] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997379] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997381] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997382] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997384] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997386] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997388] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997390] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997392] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997394] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997395] Stack: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997396] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997398] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997400] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997403] Call Trace: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997438] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997485] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997534] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997581] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997628] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997644] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997654] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997666] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997678] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997729] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997739] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997755] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997785] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997802] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997831] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997846] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997861] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997911] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997915] [] ? strlcpy+0x4a/0x60 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997931] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997946] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997961] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.997975] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998020] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998060] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998114] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998168] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998221] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998273] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998277] [] ? thread_return+0x4e/0x7d0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998328] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998332] [] ? kthread+0x9e/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998335] [] ? child_rip+0xa/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998338] [] ? kthread+0x0/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998340] [] ? child_rip+0x0/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998341] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998356] Call Trace: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998399] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998445] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998495] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998542] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998589] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998604] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998615] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998626] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998638] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998689] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998699] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998715] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998746] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998762] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998791] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998805] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998820] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998870] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998874] [] ? strlcpy+0x4a/0x60 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998890] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998905] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998920] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998935] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.998978] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999019] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999073] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999126] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999180] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999232] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999235] [] ? thread_return+0x4e/0x7d0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999287] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999290] [] ? kthread+0x9e/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999293] [] ? child_rip+0xa/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999296] [] ? kthread+0x0/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482165.999298] [] ? child_rip+0x0/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000284] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000285] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000325] CPU 7 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000325] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000362] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000363] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000366] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000371] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000372] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000374] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000376] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000377] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000379] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000381] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000383] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000384] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000388] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000390] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000391] Stack: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000392] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000394] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000396] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000398] Call Trace: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000432] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000479] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000529] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000575] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000622] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000637] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000648] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000660] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000670] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000686] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000702] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000731] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000745] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000760] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000811] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000815] [] ? strlcpy+0x4a/0x60 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000831] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000845] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000860] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000875] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000919] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.000959] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001013] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001067] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001121] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001173] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001176] [] ? thread_return+0x4e/0x7d0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001228] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001231] [] ? kthread+0x9e/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001234] [] ? child_rip+0xa/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001237] [] ? kthread+0x0/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001239] [] ? child_rip+0x0/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001240] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001255] Call Trace: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001298] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001344] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001394] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001440] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001487] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001503] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001513] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001525] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001536] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001551] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001567] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001595] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001610] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001625] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001675] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001679] [] ? strlcpy+0x4a/0x60 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001695] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001710] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001725] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001740] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001783] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001824] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001877] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001931] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.001985] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.002036] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.002040] [] ? thread_return+0x4e/0x7d0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.002092] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.002095] [] ? kthread+0x9e/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.002098] [] ? child_rip+0xa/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.002101] [] ? kthread+0x0/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482166.002103] [] ? child_rip+0x0/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.524584] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.622140] CPU 4 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.624272] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.721930] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.723787] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.737341] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.746026] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.751885] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.759856] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.767823] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.775771] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.783684] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.791580] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.800507] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.806859] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.814816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.822737] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.830746] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.840288] Stack: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.842680] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.850696] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.859189] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.868072] Call Trace: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.870992] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.878399] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.886496] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.894250] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.901737] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.909308] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.916895] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.924317] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.931431] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.938274] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.946204] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.953463] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.961769] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.968140] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.975213] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.983190] [] ? strlcpy+0x4a/0x60 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.988828] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482168.995605] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.002946] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.010114] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.017387] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.025019] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.033094] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.040478] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.047447] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.055392] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.062512] [] ? thread_return+0x4e/0x7d0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.068826] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.075780] [] ? kthread+0x9e/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.081405] [] ? child_rip+0xa/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.087055] [] ? kthread+0x0/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.095306] [] ? child_rip+0x0/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.100999] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.122458] Call Trace: Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.125326] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.132731] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.140768] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.148487] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.155984] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.163528] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.171184] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.178579] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.185720] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.192631] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.200424] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.207587] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.215919] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.222247] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.229276] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.237245] [] ? strlcpy+0x4a/0x60 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.242875] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.249659] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.257070] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.264272] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.271608] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.279151] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.287261] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.294697] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.301649] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.309602] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.316738] [] ? thread_return+0x4e/0x7d0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.323037] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.329913] [] ? kthread+0x9e/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.335512] [] ? child_rip+0xa/0x20 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.341239] [] ? kthread+0x0/0xc0 Aug 25 19:58:27 feral28.ccs.ornl.gov kernel: [1482169.346787] [] ? child_rip+0x0/0x20 Aug 25 19:59:40 feral28.ccs.ornl.gov kernel: [1482242.377510] Lustre: sultan-MDT000b: Client 5d9f30ae-edef-9a3d-c88a-aa425ee6b9cf (at 15@gni1) reconnecting Aug 25 19:59:40 feral28.ccs.ornl.gov kernel: [1482242.387920] Lustre: Skipped 8 previous similar messages Aug 25 19:59:40 feral28.ccs.ornl.gov kernel: [1482242.393743] Lustre: sultan-MDT000b: Connection restored to 62645a1e-aa58-fbd2-2d3b-02416b8b730f (at 15@gni1) Aug 25 19:59:40 feral28.ccs.ornl.gov kernel: [1482242.404409] Lustre: Skipped 8 previous similar messages Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.996899] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999899] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999900] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999943] CPU 5 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999944] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999980] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999982] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999985] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999989] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999991] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999992] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999994] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999995] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999997] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482249.999999] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000001] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000002] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000008] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000009] Stack: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000010] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000012] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000015] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000017] Call Trace: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000061] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000108] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000157] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000204] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000251] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000267] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000277] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000289] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000300] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000315] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000331] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000361] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000375] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000390] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000440] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000444] [] ? strlcpy+0x4a/0x60 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000461] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000475] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000490] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000505] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000549] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000589] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000643] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000697] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000751] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000803] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000806] [] ? thread_return+0x4e/0x7d0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000858] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000861] [] ? kthread+0x9e/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000864] [] ? child_rip+0xa/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000867] [] ? kthread+0x0/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000869] [] ? child_rip+0x0/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000870] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000885] Call Trace: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000928] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.000974] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001024] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001070] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001117] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001132] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001143] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001155] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001165] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001181] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001196] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001225] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001240] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001255] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001306] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001310] [] ? strlcpy+0x4a/0x60 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001326] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001340] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001355] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001370] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001414] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001454] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001508] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001562] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001616] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001668] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001671] [] ? thread_return+0x4e/0x7d0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001723] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001726] [] ? kthread+0x9e/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001729] [] ? child_rip+0xa/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001732] [] ? kthread+0x0/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.001734] [] ? child_rip+0x0/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003899] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003901] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003943] CPU 6 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003944] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003980] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003982] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003984] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003989] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003991] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003992] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003994] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003995] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003997] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.003999] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004001] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004003] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004008] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004010] Stack: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004011] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004013] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004015] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004018] Call Trace: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004052] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004099] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004149] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004195] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004242] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004258] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004269] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004280] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004292] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004343] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004353] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004369] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004400] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004416] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004445] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004460] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004475] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004525] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004529] [] ? strlcpy+0x4a/0x60 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004546] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004560] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004575] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004590] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004634] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004674] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004728] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004782] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004835] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004887] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004891] [] ? thread_return+0x4e/0x7d0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004942] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004946] [] ? kthread+0x9e/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004948] [] ? child_rip+0xa/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004951] [] ? kthread+0x0/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004953] [] ? child_rip+0x0/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004954] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.004970] Call Trace: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005012] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005059] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005109] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005155] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005202] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005217] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005228] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005239] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005251] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005301] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005312] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005327] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005356] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005373] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005402] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005416] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005431] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005481] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005485] [] ? strlcpy+0x4a/0x60 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005501] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005516] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005531] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005546] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005589] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005630] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005684] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005738] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005791] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005843] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005846] [] ? thread_return+0x4e/0x7d0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005898] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005902] [] ? kthread+0x9e/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005904] [] ? child_rip+0xa/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005907] [] ? kthread+0x0/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.005909] [] ? child_rip+0x0/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006898] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006899] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006939] CPU 7 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006939] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006976] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006978] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006981] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006985] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006986] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006988] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006990] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006991] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006993] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006995] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006996] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.006998] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007003] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007004] Stack: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007005] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007007] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007009] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007012] Call Trace: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007046] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007092] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007142] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007188] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007235] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007250] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007261] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007273] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007283] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007299] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007315] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007343] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007358] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007373] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007423] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007427] [] ? strlcpy+0x4a/0x60 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007443] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007458] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007473] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007488] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007532] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007580] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007634] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007687] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007741] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007793] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007796] [] ? thread_return+0x4e/0x7d0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007848] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007851] [] ? kthread+0x9e/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007854] [] ? child_rip+0xa/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007857] [] ? kthread+0x0/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007859] [] ? child_rip+0x0/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007860] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007875] Call Trace: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007918] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.007964] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008014] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008061] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008107] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008123] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008133] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008145] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008156] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008171] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008187] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008215] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008230] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008245] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008295] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008299] [] ? strlcpy+0x4a/0x60 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008315] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008330] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008344] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008359] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008403] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008443] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008497] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008551] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008605] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008656] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008660] [] ? thread_return+0x4e/0x7d0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008711] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008715] [] ? kthread+0x9e/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008718] [] ? child_rip+0xa/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008720] [] ? kthread+0x0/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482250.008723] [] ? child_rip+0x0/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.530040] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.627327] CPU 4 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.629433] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.727179] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.729061] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.742546] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.751222] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.757107] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.765011] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.772975] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.780897] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.788862] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.796732] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.805668] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.811977] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.819924] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.827898] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.835828] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.847928] Stack: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.850303] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.858205] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.866709] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.875600] Call Trace: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.878510] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.885909] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.893972] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.901604] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.909075] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.916626] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.924215] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.931566] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.938636] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.945521] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.953341] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.960479] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.968776] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.975094] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.982055] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.989974] [] ? strlcpy+0x4a/0x60 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482252.995636] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.002430] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.009841] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.017068] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.024395] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.031974] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.040101] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.047492] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.054445] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.062382] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.069466] [] ? thread_return+0x4e/0x7d0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.075790] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.082716] [] ? kthread+0x9e/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.088272] [] ? child_rip+0xa/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.093958] [] ? kthread+0x0/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.099453] [] ? child_rip+0x0/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.105103] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.126674] Call Trace: Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.129611] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.137007] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.145097] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.152686] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.160138] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.167640] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.175253] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.182577] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.189649] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.196466] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.204267] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.211396] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.219685] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.226011] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.232971] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.240915] [] ? strlcpy+0x4a/0x60 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.246553] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.253372] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.260723] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.267968] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.275330] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.282867] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.290879] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.298262] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.305215] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.313056] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.320175] [] ? thread_return+0x4e/0x7d0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.326551] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.333454] [] ? kthread+0x9e/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.339044] [] ? child_rip+0xa/0x20 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.344719] [] ? kthread+0x0/0xc0 Aug 25 19:59:51 feral28.ccs.ornl.gov kernel: [1482253.350215] [] ? child_rip+0x0/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.003513] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006513] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006515] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006558] CPU 5 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006559] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006594] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006596] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006599] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006603] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006605] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006606] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006608] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006609] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006611] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006613] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006615] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006616] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006620] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006622] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006623] Stack: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006624] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006626] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006628] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006631] Call Trace: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006675] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006721] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006771] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006818] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006865] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006881] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006891] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006904] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006914] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006929] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006945] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006975] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.006989] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007004] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007055] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007059] [] ? strlcpy+0x4a/0x60 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007075] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007090] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007104] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007119] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007163] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007203] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007257] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007311] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007365] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007417] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007420] [] ? thread_return+0x4e/0x7d0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007472] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007475] [] ? kthread+0x9e/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007478] [] ? child_rip+0xa/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007481] [] ? kthread+0x0/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007483] [] ? child_rip+0x0/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007484] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007499] Call Trace: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007542] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007589] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007638] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007685] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007731] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007747] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007758] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007769] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007780] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007795] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007811] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007840] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007855] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007869] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007920] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007924] [] ? strlcpy+0x4a/0x60 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007940] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007955] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007969] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.007984] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008028] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008068] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008122] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008176] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008230] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008282] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008285] [] ? thread_return+0x4e/0x7d0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008337] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008340] [] ? kthread+0x9e/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008343] [] ? child_rip+0xa/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008346] [] ? kthread+0x0/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.008348] [] ? child_rip+0x0/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010513] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010515] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010557] CPU 6 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010558] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010594] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010596] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010599] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010604] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010605] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010607] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010608] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010610] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010611] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010613] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010615] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010617] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010619] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010620] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010622] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010624] Stack: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010625] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010627] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010629] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010631] Call Trace: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010667] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010713] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010763] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010809] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010856] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010872] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010883] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010894] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010906] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010957] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010967] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.010983] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011012] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011028] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011057] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011072] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011087] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011137] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011141] [] ? strlcpy+0x4a/0x60 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011158] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011173] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011187] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011202] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011246] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011286] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011340] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011394] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011448] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011499] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011503] [] ? thread_return+0x4e/0x7d0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011555] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011558] [] ? kthread+0x9e/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011561] [] ? child_rip+0xa/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011563] [] ? kthread+0x0/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011566] [] ? child_rip+0x0/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011567] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011582] Call Trace: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011625] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011671] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011721] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011767] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011814] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011829] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011840] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011852] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011864] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011914] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011924] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011940] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011968] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.011985] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012013] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012027] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012042] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012093] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012096] [] ? strlcpy+0x4a/0x60 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012112] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012127] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012142] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012157] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012201] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012241] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012295] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012349] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012402] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012454] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012458] [] ? thread_return+0x4e/0x7d0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012509] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012513] [] ? kthread+0x9e/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012516] [] ? child_rip+0xa/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012518] [] ? kthread+0x0/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.012521] [] ? child_rip+0x0/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013513] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013514] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013554] CPU 7 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013555] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013591] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013593] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013596] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013600] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013602] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013603] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013605] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013607] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013608] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013611] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013613] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013614] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013616] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013618] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013620] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013621] Stack: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013622] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013623] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013626] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013628] Call Trace: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013662] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013708] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013758] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013805] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013851] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013867] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013878] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013890] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013900] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013915] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013931] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013960] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013975] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.013990] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014040] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014044] [] ? strlcpy+0x4a/0x60 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014061] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014075] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014090] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014105] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014149] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014189] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014243] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014297] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014350] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014402] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014406] [] ? thread_return+0x4e/0x7d0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014457] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014461] [] ? kthread+0x9e/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014464] [] ? child_rip+0xa/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014467] [] ? kthread+0x0/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014469] [] ? child_rip+0x0/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014470] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014485] Call Trace: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014528] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014574] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014624] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014670] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014717] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014733] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014743] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014755] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014766] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014781] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014797] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014825] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014840] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014855] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014905] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014909] [] ? strlcpy+0x4a/0x60 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014925] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014940] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014954] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.014969] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015013] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015053] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015108] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015162] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015215] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015267] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015270] [] ? thread_return+0x4e/0x7d0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015322] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015326] [] ? kthread+0x9e/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015328] [] ? child_rip+0xa/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015331] [] ? kthread+0x0/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482334.015333] [] ? child_rip+0x0/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.535921] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.636059] CPU 4 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.638173] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.735747] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.737594] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.751111] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.759729] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.765613] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.773560] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.781447] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.789412] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.797386] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.805291] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.814305] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.820675] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.828637] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.836544] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.844509] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.853929] Stack: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.856373] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.864260] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.872753] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.881585] Call Trace: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.884522] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.891919] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.900146] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.907917] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.915389] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.922985] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.930607] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.937975] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.945090] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.952045] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.959864] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.967043] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.975393] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.981808] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.988872] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482336.996815] [] ? strlcpy+0x4a/0x60 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.002436] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.009237] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.016605] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.023789] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.031115] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.038756] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.046875] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.054293] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.061244] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.069172] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.076231] [] ? thread_return+0x4e/0x7d0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.082519] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.089370] [] ? kthread+0x9e/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.094995] [] ? child_rip+0xa/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.100732] [] ? kthread+0x0/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.106312] [] ? child_rip+0x0/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.112014] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.133405] Call Trace: Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.136350] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.143754] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.151851] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.159546] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.167069] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.174664] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.182339] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.189750] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.196934] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.203830] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.211725] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.218932] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.227264] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.233599] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.240664] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.248608] [] ? strlcpy+0x4a/0x60 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.254281] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.261143] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.268536] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.275739] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.283048] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.290611] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.298745] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.306181] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.313193] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.321095] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.328178] [] ? thread_return+0x4e/0x7d0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.334451] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.341371] [] ? kthread+0x9e/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.347013] [] ? child_rip+0xa/0x20 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.352776] [] ? kthread+0x0/0xc0 Aug 25 20:01:15 feral28.ccs.ornl.gov kernel: [1482337.358296] [] ? child_rip+0x0/0x20 Aug 25 20:01:20 feral28.ccs.ornl.gov kernel: [1482342.908409] Lustre: sultan-MDT000b: haven't heard from client cc3b2647-4e02-c39c-f43e-826f0fc767f8 (at 28@gni1) in 351 seconds. I think it's dead, and I am evicting it. exp ffff880ba4e00c00, cur 1472169680 expire 1472169458 last 1472169329 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.010128] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013128] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013130] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013172] CPU 5 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013173] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013209] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013211] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013214] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013218] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013219] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013221] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013223] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013224] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013226] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013228] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013230] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013232] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013233] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013235] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013237] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013239] Stack: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013240] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013242] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013244] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013246] Call Trace: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013290] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013336] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013386] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013433] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013480] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013495] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013506] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013518] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013529] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013544] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013561] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013590] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013605] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013620] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013670] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013674] [] ? strlcpy+0x4a/0x60 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013690] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013705] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013720] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013735] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013779] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013819] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013873] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013927] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.013981] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014033] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014036] [] ? thread_return+0x4e/0x7d0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014088] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014092] [] ? kthread+0x9e/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014095] [] ? child_rip+0xa/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014097] [] ? kthread+0x0/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014100] [] ? child_rip+0x0/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014101] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014116] Call Trace: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014159] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014205] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014255] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014301] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014348] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014364] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014374] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014386] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014397] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014412] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014428] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014457] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014471] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014486] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014536] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014540] [] ? strlcpy+0x4a/0x60 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014556] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014571] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014586] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014601] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014645] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014685] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014739] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014793] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014847] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014899] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014902] [] ? thread_return+0x4e/0x7d0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014954] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014957] [] ? kthread+0x9e/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014960] [] ? child_rip+0xa/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014963] [] ? kthread+0x0/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.014965] [] ? child_rip+0x0/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017128] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017130] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017173] CPU 6 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017174] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017210] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017212] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017215] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017219] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017221] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017222] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017224] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017226] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017227] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017229] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017231] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017233] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017234] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017236] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017238] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017239] Stack: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017240] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017242] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017245] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017247] Call Trace: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017282] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017329] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017379] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017425] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017472] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017488] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017499] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017510] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017523] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017573] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017584] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017599] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017630] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017647] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017676] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017690] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017705] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017756] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017760] [] ? strlcpy+0x4a/0x60 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017776] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017791] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017806] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017820] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017864] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017905] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.017959] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018013] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018066] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018118] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018122] [] ? thread_return+0x4e/0x7d0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018174] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018177] [] ? kthread+0x9e/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018180] [] ? child_rip+0xa/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018183] [] ? kthread+0x0/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018185] [] ? child_rip+0x0/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018186] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018201] Call Trace: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018244] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018290] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018340] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018386] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018433] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018448] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018459] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018471] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018483] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018533] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018544] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018559] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018590] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018606] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018635] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018649] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018664] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018714] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018718] [] ? strlcpy+0x4a/0x60 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018734] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018749] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018764] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018779] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018823] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018863] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018917] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.018971] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.019024] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.019076] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.019080] [] ? thread_return+0x4e/0x7d0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.019132] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.019136] [] ? kthread+0x9e/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.019138] [] ? child_rip+0xa/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.019141] [] ? kthread+0x0/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.019143] [] ? child_rip+0x0/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020127] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020129] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020168] CPU 7 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020169] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020204] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020206] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020209] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020213] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020215] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020217] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020219] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020220] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020222] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020224] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020226] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020228] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020229] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020231] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020233] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020234] Stack: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020235] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020237] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020239] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020241] Call Trace: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020275] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020322] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020371] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020418] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020464] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020480] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020490] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020502] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020513] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020528] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020544] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020573] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020588] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020603] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020653] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020657] [] ? strlcpy+0x4a/0x60 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020673] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020688] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020702] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020717] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020761] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020801] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020856] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020909] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.020963] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021015] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021019] [] ? thread_return+0x4e/0x7d0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021071] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021074] [] ? kthread+0x9e/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021077] [] ? child_rip+0xa/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021080] [] ? kthread+0x0/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021082] [] ? child_rip+0x0/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021083] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021099] Call Trace: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021141] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021188] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021237] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021284] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021330] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021346] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021356] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021368] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021379] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021394] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021410] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021438] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021453] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021468] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021518] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021522] [] ? strlcpy+0x4a/0x60 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021538] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021553] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021567] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021582] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021626] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021666] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021720] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021774] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021828] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021879] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021883] [] ? thread_return+0x4e/0x7d0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021934] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021938] [] ? kthread+0x9e/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021941] [] ? child_rip+0xa/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021943] [] ? kthread+0x0/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482418.021946] [] ? child_rip+0x0/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.545485] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.642866] CPU 4 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.644972] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.742805] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.744643] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.758163] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.766805] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.772689] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.780551] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.788420] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.796324] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.804315] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.812246] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.821207] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.827559] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.835517] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.843437] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.851455] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.860962] Stack: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.863371] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.871300] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.879752] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.888593] Call Trace: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.891487] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.898908] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.906963] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.914674] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.922188] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.929714] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.937405] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.944740] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.951862] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.958774] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.966602] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.973791] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.982131] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.988519] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482420.995566] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.003475] [] ? strlcpy+0x4a/0x60 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.009113] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.015916] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.023317] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.030519] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.037802] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.045417] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.053482] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.060953] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.067921] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.075866] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.082976] [] ? thread_return+0x4e/0x7d0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.089266] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.096126] [] ? kthread+0x9e/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.101732] [] ? child_rip+0xa/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.107488] [] ? kthread+0x0/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.113025] [] ? child_rip+0x0/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.118700] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.140126] Call Trace: Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.143053] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.150476] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.158505] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.166171] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.173669] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.181228] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.188913] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.196341] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.203361] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.210290] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.218152] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.225340] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.233655] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.240018] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.247106] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.254983] [] ? strlcpy+0x4a/0x60 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.260638] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.267474] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.274797] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.282017] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.289352] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.296916] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.305006] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.312467] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.319393] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.327356] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.334424] [] ? thread_return+0x4e/0x7d0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.340670] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.347590] [] ? kthread+0x9e/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.353231] [] ? child_rip+0xa/0x20 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.358943] [] ? kthread+0x0/0xc0 Aug 25 20:02:39 feral28.ccs.ornl.gov kernel: [1482421.364506] [] ? child_rip+0x0/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.016743] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019743] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019744] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019786] CPU 5 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019787] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019823] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019825] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019828] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019832] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019834] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019836] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019837] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019839] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019840] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019842] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019844] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019846] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019849] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019851] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019852] Stack: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019853] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019855] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019857] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019859] Call Trace: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019903] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019949] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.019999] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020045] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020092] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020108] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020118] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020130] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020141] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020156] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020172] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020201] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020216] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020231] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020282] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020286] [] ? strlcpy+0x4a/0x60 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020302] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020317] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020331] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020346] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020390] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020430] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020485] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020539] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020592] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020644] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020647] [] ? thread_return+0x4e/0x7d0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020699] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020703] [] ? kthread+0x9e/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020705] [] ? child_rip+0xa/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020708] [] ? kthread+0x0/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020710] [] ? child_rip+0x0/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020712] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020727] Call Trace: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020769] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020816] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020866] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020912] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020959] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020974] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020985] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.020997] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021007] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021022] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021038] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021067] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021082] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021097] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021147] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021151] [] ? strlcpy+0x4a/0x60 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021167] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021182] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021197] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021212] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021255] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021296] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021349] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021403] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021457] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021509] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021513] [] ? thread_return+0x4e/0x7d0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021565] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021568] [] ? kthread+0x9e/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021571] [] ? child_rip+0xa/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021574] [] ? kthread+0x0/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.021576] [] ? child_rip+0x0/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023743] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023744] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023787] CPU 6 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023788] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023824] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023825] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023828] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023833] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023835] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023836] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023838] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023839] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023841] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023843] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023845] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023846] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023848] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023850] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023852] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023853] Stack: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023854] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023856] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023858] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023861] Call Trace: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023896] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023943] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.023993] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024039] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024086] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024102] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024112] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024124] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024136] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024186] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024197] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024213] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024244] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024260] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024289] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024304] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024319] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024369] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024373] [] ? strlcpy+0x4a/0x60 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024389] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024404] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024419] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024434] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024478] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024518] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024572] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024626] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024680] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024731] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024735] [] ? thread_return+0x4e/0x7d0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024787] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024790] [] ? kthread+0x9e/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024793] [] ? child_rip+0xa/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024796] [] ? kthread+0x0/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024798] [] ? child_rip+0x0/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024799] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024814] Call Trace: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024857] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024903] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024953] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.024999] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025046] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025061] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025072] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025084] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025096] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025146] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025156] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025172] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025203] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025219] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025248] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025262] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025277] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025328] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025332] [] ? strlcpy+0x4a/0x60 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025348] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025363] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025377] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025392] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025436] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025476] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025530] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025584] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025637] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025689] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025693] [] ? thread_return+0x4e/0x7d0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025745] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025748] [] ? kthread+0x9e/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025751] [] ? child_rip+0xa/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025753] [] ? kthread+0x0/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.025755] [] ? child_rip+0x0/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026742] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026743] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026783] CPU 7 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026784] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026819] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026821] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026824] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026828] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026829] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026831] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026833] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026834] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026836] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026838] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026840] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026842] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026843] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026845] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026847] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026848] Stack: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026849] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026851] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026853] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026856] Call Trace: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026889] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026936] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.026986] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027032] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027079] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027094] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027105] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027117] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027128] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027143] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027159] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027188] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027202] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027217] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027268] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027272] [] ? strlcpy+0x4a/0x60 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027288] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027303] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027317] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027332] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027376] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027416] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027470] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027524] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027578] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027630] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027633] [] ? thread_return+0x4e/0x7d0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027685] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027688] [] ? kthread+0x9e/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027691] [] ? child_rip+0xa/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027694] [] ? kthread+0x0/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027696] [] ? child_rip+0x0/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027697] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027712] Call Trace: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027755] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027801] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027851] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027897] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027944] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027960] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027970] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027982] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.027992] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028007] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028023] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028052] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028067] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028082] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028132] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028136] [] ? strlcpy+0x4a/0x60 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028152] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028167] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028181] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028196] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028240] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028280] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028334] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028388] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028442] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028493] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028497] [] ? thread_return+0x4e/0x7d0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028549] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028552] [] ? kthread+0x9e/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028555] [] ? child_rip+0xa/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028558] [] ? kthread+0x0/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482502.028560] [] ? child_rip+0x0/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.531942] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.629021] CPU 4 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.631101] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.728405] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.730260] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.743668] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.752301] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.758160] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.766037] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.773940] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.781810] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.789742] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.797628] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.806538] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.812872] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.820717] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.828561] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.836455] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.845884] Stack: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.848260] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.856165] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.864624] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.873438] Call Trace: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.876375] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.883832] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.891861] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.899476] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.906956] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.914506] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.922112] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.929455] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.936508] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.943377] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.951230] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.958368] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.966604] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.972914] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.979944] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.987835] [] ? strlcpy+0x4a/0x60 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482504.993448] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.000336] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.007773] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.014915] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.022207] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.029785] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.037878] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.045280] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.052264] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.060165] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.067225] [] ? thread_return+0x4e/0x7d0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.073540] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.080450] [] ? kthread+0x9e/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.086030] [] ? child_rip+0xa/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.091733] [] ? kthread+0x0/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.097203] [] ? child_rip+0x0/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.102844] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.124452] Call Trace: Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.127388] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.134819] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.142830] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.150462] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.160559] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.168009] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.175622] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.182929] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.189982] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.196895] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.204696] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.211885] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.220165] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.226484] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.233479] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.241371] [] ? strlcpy+0x4a/0x60 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.247009] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.253889] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.261256] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.268467] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.275742] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.283337] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.291439] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.298855] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.305764] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.313676] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.320769] [] ? thread_return+0x4e/0x7d0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.327101] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.334029] [] ? kthread+0x9e/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.339628] [] ? child_rip+0xa/0x20 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.345356] [] ? kthread+0x0/0xc0 Aug 25 20:04:03 feral28.ccs.ornl.gov kernel: [1482505.350832] [] ? child_rip+0x0/0x20 Aug 25 20:04:04 feral28.ccs.ornl.gov kernel: [1482507.032251] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 20:04:04 feral28.ccs.ornl.gov kernel: [1482507.043330] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 32 previous similar messages Aug 25 20:04:04 feral28.ccs.ornl.gov kernel: [1482507.054478] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.101@o2ib1 (212): c: 63, oc: 0, rc: 63 Aug 25 20:04:04 feral28.ccs.ornl.gov kernel: [1482507.067951] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 32 previous similar messages Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.023357] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026357] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026359] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026401] CPU 5 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026402] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026439] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026440] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026443] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026448] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026449] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026451] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026452] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026454] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026455] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026457] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026459] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026461] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026462] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026464] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026466] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026468] Stack: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026468] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026470] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026473] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026475] Call Trace: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026519] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026565] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026615] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026662] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026708] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026724] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026735] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026747] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026758] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026773] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026789] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026818] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026833] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026848] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026898] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026902] [] ? strlcpy+0x4a/0x60 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026918] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026933] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026948] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.026963] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027006] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027047] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027101] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027155] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027209] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027260] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027264] [] ? thread_return+0x4e/0x7d0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027316] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027319] [] ? kthread+0x9e/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027322] [] ? child_rip+0xa/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027325] [] ? kthread+0x0/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027327] [] ? child_rip+0x0/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027328] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027343] Call Trace: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027386] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027432] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027482] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027528] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027575] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027590] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027601] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027613] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027623] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027638] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027654] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027683] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027697] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027712] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027762] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027766] [] ? strlcpy+0x4a/0x60 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027782] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027797] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027812] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027827] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027871] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027911] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.027965] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.028019] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.028073] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.028124] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.028128] [] ? thread_return+0x4e/0x7d0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.028180] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.028183] [] ? kthread+0x9e/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.028186] [] ? child_rip+0xa/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.028189] [] ? kthread+0x0/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.028191] [] ? child_rip+0x0/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030357] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030359] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030402] CPU 6 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030402] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030438] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030440] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030443] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030447] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030449] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030451] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030452] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030454] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030455] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030457] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030459] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030461] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030462] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030464] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030466] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030467] Stack: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030468] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030470] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030472] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030475] Call Trace: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030510] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030557] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030607] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030653] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030700] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030716] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030726] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030738] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030750] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030801] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030811] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030827] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030858] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030874] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030903] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030918] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030933] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030983] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.030987] [] ? strlcpy+0x4a/0x60 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031004] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031018] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031033] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031048] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031092] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031132] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031186] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031240] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031294] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031346] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031349] [] ? thread_return+0x4e/0x7d0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031401] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031404] [] ? kthread+0x9e/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031407] [] ? child_rip+0xa/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031410] [] ? kthread+0x0/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031412] [] ? child_rip+0x0/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031413] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031428] Call Trace: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031471] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031518] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031567] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031614] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031660] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031676] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031686] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031698] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031710] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031761] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031771] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031786] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031817] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031834] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031862] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031877] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031892] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031942] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031946] [] ? strlcpy+0x4a/0x60 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031962] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031977] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.031992] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032006] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032050] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032091] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032144] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032199] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032252] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032304] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032307] [] ? thread_return+0x4e/0x7d0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032359] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032362] [] ? kthread+0x9e/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032365] [] ? child_rip+0xa/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032368] [] ? kthread+0x0/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.032370] [] ? child_rip+0x0/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033357] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033358] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033398] CPU 7 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033398] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033434] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033436] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033439] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033443] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033444] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033446] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033448] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033449] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033451] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033453] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033455] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033457] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033460] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033462] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033463] Stack: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033464] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033466] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033468] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033471] Call Trace: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033504] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033551] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033600] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033647] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033694] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033709] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033720] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033732] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033742] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033757] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033773] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033802] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033817] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033832] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033882] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033886] [] ? strlcpy+0x4a/0x60 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033902] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033917] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033932] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033947] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.033990] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034031] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034084] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034139] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034192] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034244] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034247] [] ? thread_return+0x4e/0x7d0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034299] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034302] [] ? kthread+0x9e/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034305] [] ? child_rip+0xa/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034308] [] ? kthread+0x0/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034310] [] ? child_rip+0x0/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034311] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034327] Call Trace: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034369] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034416] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034465] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034512] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034558] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034574] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034584] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034596] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034607] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034622] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034638] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034667] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034681] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034696] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034746] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034750] [] ? strlcpy+0x4a/0x60 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034766] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034781] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034796] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034811] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034854] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034895] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.034949] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.035003] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.035056] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.035108] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.035111] [] ? thread_return+0x4e/0x7d0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.035163] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.035166] [] ? kthread+0x9e/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.035169] [] ? child_rip+0xa/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.035172] [] ? kthread+0x0/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482586.035174] [] ? child_rip+0x0/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.556136] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.653413] CPU 4 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.655546] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.753040] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.754879] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.768407] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.777066] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.782977] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.790812] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.798733] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.806690] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.814662] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.822566] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.831494] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.837776] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.845718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.853621] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.861552] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.871025] Stack: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.873434] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.881364] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.889882] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.901401] Call Trace: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.904348] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.911752] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.919877] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.927501] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.934954] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.942524] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.950171] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.957496] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.964620] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.971541] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.979453] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.986566] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482588.994933] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.001321] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.008331] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.016233] [] ? strlcpy+0x4a/0x60 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.021888] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.028654] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.036066] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.043251] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.050534] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.058157] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.066197] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.073633] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.080618] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.088475] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.095509] [] ? thread_return+0x4e/0x7d0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.101807] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.108683] [] ? kthread+0x9e/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.114325] [] ? child_rip+0xa/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.119992] [] ? kthread+0x0/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.125505] [] ? child_rip+0x0/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.131190] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.152725] Call Trace: Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.155618] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.163049] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.171097] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.178686] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.186123] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.193554] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.201157] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.208457] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.215493] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.222336] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.230127] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.237275] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.245581] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.251872] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.258841] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.266684] [] ? strlcpy+0x4a/0x60 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.272320] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.279140] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.286515] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.293683] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.300948] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.308503] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.316612] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.324030] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.330981] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.338910] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.345986] [] ? thread_return+0x4e/0x7d0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.352318] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.359220] [] ? kthread+0x9e/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.364748] [] ? child_rip+0xa/0x20 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.370391] [] ? kthread+0x0/0xc0 Aug 25 20:05:27 feral28.ccs.ornl.gov kernel: [1482589.375876] [] ? child_rip+0x0/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.029972] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.032972] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.032973] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033015] CPU 5 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033016] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033053] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033054] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033057] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033061] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033063] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033065] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033066] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033068] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033069] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033071] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033073] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033075] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033076] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033078] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033080] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033082] Stack: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033082] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033085] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033087] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033089] Call Trace: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033133] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033180] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033229] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033276] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033323] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033338] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033349] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033361] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033372] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033387] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033403] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033432] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033447] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033462] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033513] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033517] [] ? strlcpy+0x4a/0x60 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033533] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033548] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033562] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033577] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033621] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033662] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033716] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033770] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033824] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033876] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033879] [] ? thread_return+0x4e/0x7d0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033931] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033934] [] ? kthread+0x9e/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033937] [] ? child_rip+0xa/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033940] [] ? kthread+0x0/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033942] [] ? child_rip+0x0/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033943] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.033958] Call Trace: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034001] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034047] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034097] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034143] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034190] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034206] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034216] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034228] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034239] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034254] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034270] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034299] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034313] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034328] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034378] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034382] [] ? strlcpy+0x4a/0x60 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034398] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034413] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034428] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034443] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034486] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034527] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034581] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034635] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034689] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034741] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034744] [] ? thread_return+0x4e/0x7d0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034796] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034799] [] ? kthread+0x9e/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034802] [] ? child_rip+0xa/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034805] [] ? kthread+0x0/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.034807] [] ? child_rip+0x0/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.036972] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.036973] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037016] CPU 6 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037017] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037053] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037054] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037057] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037062] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037063] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037065] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037066] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037068] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037069] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037071] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037073] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037075] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037076] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037078] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037080] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037081] Stack: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037082] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037084] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037086] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037089] Call Trace: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037124] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037171] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037221] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037268] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037314] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037330] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037341] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037352] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037364] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037415] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037425] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037441] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037472] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037488] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037517] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037532] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037546] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037597] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037601] [] ? strlcpy+0x4a/0x60 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037617] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037632] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037647] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037661] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037705] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037745] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037800] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037854] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037907] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037959] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.037963] [] ? thread_return+0x4e/0x7d0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038014] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038018] [] ? kthread+0x9e/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038021] [] ? child_rip+0xa/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038023] [] ? kthread+0x0/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038026] [] ? child_rip+0x0/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038027] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038042] Call Trace: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038085] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038131] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038181] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038228] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038274] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038290] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038301] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038312] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038324] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038374] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038385] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038400] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038429] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038445] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038474] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038489] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038504] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038554] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038558] [] ? strlcpy+0x4a/0x60 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038574] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038589] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038603] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038618] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038662] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038702] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038756] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038810] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038864] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038915] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038919] [] ? thread_return+0x4e/0x7d0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038971] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038974] [] ? kthread+0x9e/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038977] [] ? child_rip+0xa/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038980] [] ? kthread+0x0/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.038982] [] ? child_rip+0x0/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.039971] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.039972] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040014] CPU 7 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040015] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040052] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040053] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040056] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040060] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040062] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040063] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040065] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040066] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040068] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040070] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040072] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040074] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040075] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040077] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040079] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040080] Stack: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040081] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040083] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040086] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040088] Call Trace: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040122] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040169] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040219] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040265] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040312] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040327] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040338] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040350] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040361] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040376] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040392] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040421] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040436] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040451] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040501] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040505] [] ? strlcpy+0x4a/0x60 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040521] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040536] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040551] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040565] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040609] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040650] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040704] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040757] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040811] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040862] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040866] [] ? thread_return+0x4e/0x7d0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040918] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040921] [] ? kthread+0x9e/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040924] [] ? child_rip+0xa/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040926] [] ? kthread+0x0/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040929] [] ? child_rip+0x0/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040930] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040945] Call Trace: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.040987] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041034] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041084] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041130] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041177] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041192] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041203] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041215] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041225] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041240] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041256] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041285] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041300] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041315] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041365] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041369] [] ? strlcpy+0x4a/0x60 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041385] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041400] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041414] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041429] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041473] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041513] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041567] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041621] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041674] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041726] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041729] [] ? thread_return+0x4e/0x7d0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041781] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041784] [] ? kthread+0x9e/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041787] [] ? child_rip+0xa/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041790] [] ? kthread+0x0/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482670.041792] [] ? child_rip+0x0/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.544552] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.641761] CPU 4 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.643858] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.743798] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.745635] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.759008] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.767675] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.773517] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.781385] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.789265] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.797141] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.805039] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.812883] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.821731] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.828084] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.835961] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.843839] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.851745] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.861181] Stack: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.863592] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.871505] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.879999] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.888785] Call Trace: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.891680] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.899094] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.907097] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.914729] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.922191] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.929693] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.937270] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.944553] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.951580] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.958432] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.966233] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.973394] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.981650] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.987958] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482672.994929] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.002804] [] ? strlcpy+0x4a/0x60 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.008458] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.015313] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.022653] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.029847] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.037104] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.044692] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.052758] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.060194] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.067232] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.075109] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.082184] [] ? thread_return+0x4e/0x7d0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.088497] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.095410] [] ? kthread+0x9e/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.101008] [] ? child_rip+0xa/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.106651] [] ? kthread+0x0/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.112197] [] ? child_rip+0x0/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.117863] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.139488] Call Trace: Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.142371] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.149795] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.157780] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.165378] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.172847] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.180367] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.187936] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.195254] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.202263] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.209115] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.216951] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.224148] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.232385] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.238670] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.245646] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.253523] [] ? strlcpy+0x4a/0x60 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.259151] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.266022] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.273415] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.280608] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.287910] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.295533] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.303615] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.311034] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.317950] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.325887] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.333024] [] ? thread_return+0x4e/0x7d0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.339337] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.346249] [] ? kthread+0x9e/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.351803] [] ? child_rip+0xa/0x20 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.357473] [] ? kthread+0x0/0xc0 Aug 25 20:06:51 feral28.ccs.ornl.gov kernel: [1482673.362958] [] ? child_rip+0x0/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.036579] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039579] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039581] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039623] CPU 5 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039624] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039660] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039662] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039665] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039669] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039671] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039673] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039674] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039676] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039677] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039679] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039681] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039683] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039685] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039686] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039688] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039690] Stack: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039691] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039693] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039695] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039698] Call Trace: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039741] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039788] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039837] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039884] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039931] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039946] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039957] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039969] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039980] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.039995] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040011] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040040] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040055] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040070] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040120] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040125] [] ? strlcpy+0x4a/0x60 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040141] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040156] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040171] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040186] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040230] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040270] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040324] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040379] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040432] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040484] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040487] [] ? thread_return+0x4e/0x7d0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040539] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040543] [] ? kthread+0x9e/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040545] [] ? child_rip+0xa/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040548] [] ? kthread+0x0/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040550] [] ? child_rip+0x0/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040551] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040567] Call Trace: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040609] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040656] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040705] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040752] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040798] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040814] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040824] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040836] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040847] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040862] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040878] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040910] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040925] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040940] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040990] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.040994] [] ? strlcpy+0x4a/0x60 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041010] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041025] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041040] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041054] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041098] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041139] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041193] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041247] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041300] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041352] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041356] [] ? thread_return+0x4e/0x7d0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041408] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041411] [] ? kthread+0x9e/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041414] [] ? child_rip+0xa/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041417] [] ? kthread+0x0/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.041419] [] ? child_rip+0x0/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043579] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043581] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043624] CPU 6 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043624] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043660] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043662] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043665] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043669] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043671] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043673] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043674] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043676] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043677] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043679] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043681] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043683] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043686] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043688] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043690] Stack: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043691] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043693] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043695] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043698] Call Trace: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043733] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043779] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043829] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043876] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043922] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043938] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043949] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043961] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.043973] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044024] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044034] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044050] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044080] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044097] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044126] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044140] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044155] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044206] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044210] [] ? strlcpy+0x4a/0x60 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044226] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044241] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044256] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044270] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044314] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044354] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044409] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044463] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044516] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044568] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044572] [] ? thread_return+0x4e/0x7d0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044624] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044627] [] ? kthread+0x9e/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044630] [] ? child_rip+0xa/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044633] [] ? kthread+0x0/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044635] [] ? child_rip+0x0/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044636] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044651] Call Trace: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044694] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044740] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044790] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044836] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044883] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044899] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044909] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044921] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044933] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044983] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.044993] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045009] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045037] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045054] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045082] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045097] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045112] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045162] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045166] [] ? strlcpy+0x4a/0x60 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045182] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045197] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045212] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045226] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045270] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045310] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045364] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045418] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045472] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045524] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045527] [] ? thread_return+0x4e/0x7d0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045579] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045582] [] ? kthread+0x9e/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045585] [] ? child_rip+0xa/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045588] [] ? kthread+0x0/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.045590] [] ? child_rip+0x0/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046578] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046579] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046619] CPU 7 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046620] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046655] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046657] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046660] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046664] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046666] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046667] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046669] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046670] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046672] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046674] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046676] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046677] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046679] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046681] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046683] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046684] Stack: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046685] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046687] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046689] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046692] Call Trace: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046726] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046772] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046822] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046868] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046915] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046930] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046941] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046953] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046963] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046979] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.046995] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047024] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047038] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047053] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047104] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047107] [] ? strlcpy+0x4a/0x60 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047124] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047138] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047153] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047168] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047212] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047252] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047306] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047360] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047413] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047465] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047468] [] ? thread_return+0x4e/0x7d0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047520] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047523] [] ? kthread+0x9e/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047526] [] ? child_rip+0xa/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047529] [] ? kthread+0x0/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047531] [] ? child_rip+0x0/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047532] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047547] Call Trace: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047590] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047636] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047686] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047733] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047779] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047795] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047805] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047817] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047828] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047843] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047859] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047887] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047902] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047917] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047967] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047971] [] ? strlcpy+0x4a/0x60 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.047987] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048002] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048017] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048031] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048075] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048115] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048169] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048223] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048277] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048329] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048332] [] ? thread_return+0x4e/0x7d0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048384] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048387] [] ? kthread+0x9e/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048390] [] ? child_rip+0xa/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048392] [] ? kthread+0x0/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482754.048395] [] ? child_rip+0x0/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.576772] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.674031] CPU 4 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.676173] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.773954] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.775861] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.789390] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.798169] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.804063] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.811993] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.819949] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.827880] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.835801] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.843767] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.852781] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.859064] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.867029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.875027] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.882915] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.892378] Stack: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.894780] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.902710] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.911239] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.920088] Call Trace: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.922990] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.930343] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.938422] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.946097] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.953613] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.961175] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.968849] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.976217] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.983270] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.990157] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482756.998027] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.005260] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.013662] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.020032] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.027079] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.034963] [] ? strlcpy+0x4a/0x60 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.040573] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.047341] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.054750] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.062030] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.069307] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.076852] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.084925] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.092310] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.099261] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.107188] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.114257] [] ? thread_return+0x4e/0x7d0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.120560] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.127499] [] ? kthread+0x9e/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.133132] [] ? child_rip+0xa/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.138851] [] ? kthread+0x0/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.144416] [] ? child_rip+0x0/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.150134] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.171629] Call Trace: Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.174548] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.181953] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.190026] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.197805] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.205337] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.212838] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.220469] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.227827] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.234908] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.241812] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.249638] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.256905] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.265271] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.271658] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.278681] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.286633] [] ? strlcpy+0x4a/0x60 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.292271] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.299108] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.306458] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.313669] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.320942] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.328531] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.336624] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.344075] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.351036] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.358904] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.366006] [] ? thread_return+0x4e/0x7d0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.372380] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.379300] [] ? kthread+0x9e/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.384923] [] ? child_rip+0xa/0x20 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.390697] [] ? kthread+0x0/0xc0 Aug 25 20:08:15 feral28.ccs.ornl.gov kernel: [1482757.396209] [] ? child_rip+0x0/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.043186] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046186] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046188] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046230] CPU 5 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046231] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046268] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046270] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046273] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046277] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046279] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046281] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046282] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046284] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046286] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046288] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046290] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046291] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046295] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046297] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046298] Stack: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046299] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046301] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046303] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046306] Call Trace: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046350] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046396] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046446] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046493] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046539] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046555] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046566] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046578] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046589] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046604] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046620] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046649] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046664] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046679] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046729] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046733] [] ? strlcpy+0x4a/0x60 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046749] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046764] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046779] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046794] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046838] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046878] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046932] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.046986] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047039] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047091] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047094] [] ? thread_return+0x4e/0x7d0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047146] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047150] [] ? kthread+0x9e/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047153] [] ? child_rip+0xa/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047155] [] ? kthread+0x0/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047158] [] ? child_rip+0x0/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047159] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047174] Call Trace: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047217] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047263] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047313] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047359] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047405] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047421] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047431] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047443] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047454] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047469] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047485] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047514] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047528] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047543] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047593] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047597] [] ? strlcpy+0x4a/0x60 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047613] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047628] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047643] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047658] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047702] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047742] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047796] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047850] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047904] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047955] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.047959] [] ? thread_return+0x4e/0x7d0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.048011] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.048014] [] ? kthread+0x9e/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.048017] [] ? child_rip+0xa/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.048020] [] ? kthread+0x0/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.048022] [] ? child_rip+0x0/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050187] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050188] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050231] CPU 6 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050232] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050268] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050270] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050273] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050278] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050279] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050281] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050282] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050284] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050286] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050288] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050290] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050291] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050295] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050297] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050299] Stack: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050300] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050302] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050304] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050306] Call Trace: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050341] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050388] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050438] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050484] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050531] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050547] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050558] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050569] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050582] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050632] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050643] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050658] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050690] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050706] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050735] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050749] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050764] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050814] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050818] [] ? strlcpy+0x4a/0x60 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050835] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050849] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050864] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050879] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050923] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.050963] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051018] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051072] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051125] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051177] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051180] [] ? thread_return+0x4e/0x7d0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051232] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051236] [] ? kthread+0x9e/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051239] [] ? child_rip+0xa/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051241] [] ? kthread+0x0/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051244] [] ? child_rip+0x0/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051245] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051260] Call Trace: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051303] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051349] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051399] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051445] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051492] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051508] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051518] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051530] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051542] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051592] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051602] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051617] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051646] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051663] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051691] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051706] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051721] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051771] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051775] [] ? strlcpy+0x4a/0x60 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051791] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051806] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051820] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051835] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051879] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051927] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.051981] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.052035] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.052089] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.052141] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.052145] [] ? thread_return+0x4e/0x7d0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.052197] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.052200] [] ? kthread+0x9e/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.052203] [] ? child_rip+0xa/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.052206] [] ? kthread+0x0/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.052208] [] ? child_rip+0x0/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053186] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053187] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053227] CPU 7 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053227] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053263] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053265] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053267] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053271] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053273] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053275] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053276] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053278] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053279] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053281] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053283] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053284] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053286] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053288] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053290] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053291] Stack: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053292] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053294] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053296] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053299] Call Trace: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053332] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053379] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053429] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053475] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053522] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053537] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053548] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053560] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053571] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053586] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053602] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053631] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053645] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053660] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053710] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053714] [] ? strlcpy+0x4a/0x60 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053731] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053745] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053760] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053775] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053819] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053859] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053913] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.053967] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054020] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054072] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054076] [] ? thread_return+0x4e/0x7d0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054127] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054131] [] ? kthread+0x9e/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054134] [] ? child_rip+0xa/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054136] [] ? kthread+0x0/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054139] [] ? child_rip+0x0/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054140] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054155] Call Trace: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054197] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054244] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054294] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054340] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054387] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054402] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054413] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054425] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054435] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054450] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054466] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054495] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054510] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054525] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054575] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054579] [] ? strlcpy+0x4a/0x60 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054595] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054610] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054625] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054640] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054683] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054724] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054778] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054832] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054885] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054937] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054941] [] ? thread_return+0x4e/0x7d0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054992] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054996] [] ? kthread+0x9e/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.054998] [] ? child_rip+0xa/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.055001] [] ? kthread+0x0/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482838.055003] [] ? child_rip+0x0/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.580876] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.678248] CPU 4 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.680337] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.778283] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.780165] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.793596] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.802238] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.808133] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.816087] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.824088] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.832053] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.839897] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.847828] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.856737] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.863105] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.871020] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.878907] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.886819] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.896342] Stack: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.898737] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.906597] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.915143] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.923990] Call Trace: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.926920] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.934334] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.942414] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.950133] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.957656] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.965279] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.972918] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.980269] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.987416] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482840.994347] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.002262] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.009493] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.017782] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.024214] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.031276] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.039144] [] ? strlcpy+0x4a/0x60 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.044782] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.051566] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.058879] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.066057] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.073401] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.080920] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.089038] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.096457] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.103329] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.111215] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.118264] [] ? thread_return+0x4e/0x7d0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.124629] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.131560] [] ? kthread+0x9e/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.137185] [] ? child_rip+0xa/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.142937] [] ? kthread+0x0/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.148442] [] ? child_rip+0x0/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.154187] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.175671] Call Trace: Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.178557] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.185979] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.194016] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.201727] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.209250] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.216794] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.224451] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.231758] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.238836] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.245751] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.253595] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.260767] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.269062] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.275486] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.285229] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.293147] [] ? strlcpy+0x4a/0x60 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.298819] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.305593] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.312988] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.320232] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.327526] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.335088] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.343136] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.350529] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.357488] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.365339] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.372466] [] ? thread_return+0x4e/0x7d0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.378738] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.385588] [] ? kthread+0x9e/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.391230] [] ? child_rip+0xa/0x20 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.396968] [] ? kthread+0x0/0xc0 Aug 25 20:09:39 feral28.ccs.ornl.gov kernel: [1482841.402463] [] ? child_rip+0x0/0x20 Aug 25 20:09:44 feral28.ccs.ornl.gov kernel: [1482846.477141] Lustre: sultan-MDT000b: Client 5d9f30ae-edef-9a3d-c88a-aa425ee6b9cf (at 15@gni1) reconnecting Aug 25 20:09:44 feral28.ccs.ornl.gov kernel: [1482846.487516] Lustre: Skipped 11 previous similar messages Aug 25 20:09:44 feral28.ccs.ornl.gov kernel: [1482846.493429] Lustre: sultan-MDT000b: Connection restored to 62645a1e-aa58-fbd2-2d3b-02416b8b730f (at 15@gni1) Aug 25 20:09:44 feral28.ccs.ornl.gov kernel: [1482846.504062] Lustre: Skipped 13 previous similar messages Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.049794] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052794] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052795] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052838] CPU 5 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052839] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052875] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052877] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052881] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052885] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052886] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052888] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052890] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052892] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052894] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052896] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052898] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052899] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052902] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052904] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052905] Stack: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052906] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052908] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052910] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052913] Call Trace: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.052957] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053004] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053054] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053100] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053147] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053163] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053174] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053186] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053196] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053212] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053228] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053257] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053272] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053287] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053337] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053341] [] ? strlcpy+0x4a/0x60 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053358] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053373] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053387] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053402] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053446] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053486] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053541] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053595] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053648] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053700] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053704] [] ? thread_return+0x4e/0x7d0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053755] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053759] [] ? kthread+0x9e/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053761] [] ? child_rip+0xa/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053764] [] ? kthread+0x0/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053766] [] ? child_rip+0x0/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053768] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053783] Call Trace: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053825] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053872] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053921] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.053968] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054015] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054030] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054041] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054053] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054063] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054078] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054094] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054123] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054138] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054153] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054203] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054207] [] ? strlcpy+0x4a/0x60 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054223] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054238] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054252] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054267] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054311] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054351] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054405] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054459] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054513] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054565] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054568] [] ? thread_return+0x4e/0x7d0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054620] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054623] [] ? kthread+0x9e/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054626] [] ? child_rip+0xa/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054629] [] ? kthread+0x0/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.054631] [] ? child_rip+0x0/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056794] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056796] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056839] CPU 6 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056839] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056876] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056878] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056881] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056885] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056887] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056888] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056890] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056892] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056893] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056896] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056897] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056899] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056902] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056904] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056906] Stack: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056906] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056909] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056911] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056913] Call Trace: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056948] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.056994] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057044] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057091] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057138] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057153] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057164] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057176] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057188] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057239] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057249] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057265] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057296] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057313] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057341] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057356] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057371] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057421] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057425] [] ? strlcpy+0x4a/0x60 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057441] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057456] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057471] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057486] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057529] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057570] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057624] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057678] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057731] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057784] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057787] [] ? thread_return+0x4e/0x7d0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057838] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057842] [] ? kthread+0x9e/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057845] [] ? child_rip+0xa/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057847] [] ? kthread+0x0/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057850] [] ? child_rip+0x0/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057851] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057866] Call Trace: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057909] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.057955] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058005] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058051] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058098] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058113] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058124] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058135] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058147] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058198] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058208] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058224] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058255] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058271] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058299] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058314] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058329] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058379] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058383] [] ? strlcpy+0x4a/0x60 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058399] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058414] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058429] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058444] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058487] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058528] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058582] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058636] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058689] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058741] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058745] [] ? thread_return+0x4e/0x7d0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058796] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058800] [] ? kthread+0x9e/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058802] [] ? child_rip+0xa/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058805] [] ? kthread+0x0/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.058807] [] ? child_rip+0x0/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059793] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059795] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059835] CPU 7 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059836] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059872] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059874] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059877] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059881] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059882] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059884] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059886] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059887] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059889] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059891] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059893] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059895] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059897] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059898] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059900] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059901] Stack: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059902] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059904] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059906] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059909] Call Trace: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059943] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.059989] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060039] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060086] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060132] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060148] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060158] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060170] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060181] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060196] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060212] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060241] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060256] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060271] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060321] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060325] [] ? strlcpy+0x4a/0x60 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060341] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060356] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060371] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060385] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060429] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060469] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060524] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060577] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060630] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060682] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060686] [] ? thread_return+0x4e/0x7d0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060737] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060741] [] ? kthread+0x9e/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060743] [] ? child_rip+0xa/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060746] [] ? kthread+0x0/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060748] [] ? child_rip+0x0/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060750] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060765] Call Trace: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060807] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060853] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060903] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060950] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.060996] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061012] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061022] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061034] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061045] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061060] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061076] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061104] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061119] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061134] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061184] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061188] [] ? strlcpy+0x4a/0x60 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061204] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061219] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061233] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061248] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061292] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061332] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061386] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061440] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061494] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061546] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061549] [] ? thread_return+0x4e/0x7d0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061601] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061604] [] ? kthread+0x9e/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061607] [] ? child_rip+0xa/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061609] [] ? kthread+0x0/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482922.061611] [] ? child_rip+0x0/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.584771] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.682092] CPU 4 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.684233] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.781858] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.783757] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.797181] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.805926] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.811785] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.819742] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.827680] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.835689] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.843627] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.851627] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.860598] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.866940] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.874819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.882655] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.890559] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.900050] Stack: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.902441] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.910363] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.918935] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.927792] Call Trace: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.930730] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.938151] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.946250] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.953899] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.961448] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.969009] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.976682] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.984093] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.991148] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482924.998095] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.006001] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.015868] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.024201] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.030527] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.037609] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.045545] [] ? strlcpy+0x4a/0x60 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.051139] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.057940] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.065291] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.072473] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.079801] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.087382] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.095517] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.102942] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.109912] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.117797] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.124865] [] ? thread_return+0x4e/0x7d0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.131231] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.138117] [] ? kthread+0x9e/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.143791] [] ? child_rip+0xa/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.149495] [] ? kthread+0x0/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.155051] [] ? child_rip+0x0/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.160830] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.182298] Call Trace: Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.185183] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.192552] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.200633] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.208362] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.215876] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.223420] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.231086] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.238514] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.245654] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.252593] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.260455] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.267653] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.275967] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.282303] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.289349] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.297285] [] ? strlcpy+0x4a/0x60 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.302974] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.309827] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.317257] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.324422] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.331690] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.339288] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.347378] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.354814] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.361816] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.369693] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.376813] [] ? thread_return+0x4e/0x7d0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.383189] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.390135] [] ? kthread+0x9e/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.395750] [] ? child_rip+0xa/0x20 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.401488] [] ? kthread+0x0/0xc0 Aug 25 20:11:03 feral28.ccs.ornl.gov kernel: [1482925.407051] [] ? child_rip+0x0/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.056402] BUG: soft lockup - CPU#4 stuck for 68s! [mdt01_002:21487] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059402] BUG: soft lockup - CPU#5 stuck for 68s! [mdt01_007:17158] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059403] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059446] CPU 5 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059447] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059483] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059484] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059487] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059492] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059493] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059495] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059497] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059499] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059500] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059502] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059504] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059506] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059508] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059509] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059511] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059513] Stack: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059513] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059515] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059517] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059520] Call Trace: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059564] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059610] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059660] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059707] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059753] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059769] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059780] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059792] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059803] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059818] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059834] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059863] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059878] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059893] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059943] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059948] [] ? strlcpy+0x4a/0x60 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059964] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059979] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.059993] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060008] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060052] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060092] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060146] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060201] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060255] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060307] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060310] [] ? thread_return+0x4e/0x7d0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060362] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060366] [] ? kthread+0x9e/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060368] [] ? child_rip+0xa/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060371] [] ? kthread+0x0/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060373] [] ? child_rip+0x0/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060375] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060390] Call Trace: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060433] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060479] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060529] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060575] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060622] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060637] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060648] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060660] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060670] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060685] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060701] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060730] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060745] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060760] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060810] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060814] [] ? strlcpy+0x4a/0x60 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060830] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060845] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060860] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060874] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060918] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.060958] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061012] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061066] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061120] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061172] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061175] [] ? thread_return+0x4e/0x7d0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061227] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061230] [] ? kthread+0x9e/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061233] [] ? child_rip+0xa/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061236] [] ? kthread+0x0/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.061238] [] ? child_rip+0x0/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063402] BUG: soft lockup - CPU#6 stuck for 68s! [mdt01_000:21485] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063403] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063446] CPU 6 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063447] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063484] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063486] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063489] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063494] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063495] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063497] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063498] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063500] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063502] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063504] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063506] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063508] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063511] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063513] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063515] Stack: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063515] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063517] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063519] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063522] Call Trace: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063557] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063603] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063653] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063700] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063746] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063762] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063773] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063784] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063797] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063847] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063858] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063873] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063904] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063921] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063949] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063964] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.063979] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064030] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064034] [] ? strlcpy+0x4a/0x60 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064050] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064065] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064080] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064094] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064138] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064178] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064233] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064287] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064340] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064392] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064396] [] ? thread_return+0x4e/0x7d0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064447] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064451] [] ? kthread+0x9e/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064454] [] ? child_rip+0xa/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064456] [] ? kthread+0x0/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064458] [] ? child_rip+0x0/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064460] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064475] Call Trace: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064518] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064564] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064614] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064660] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064707] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064722] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064733] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064744] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064756] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064807] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064817] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064833] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064864] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064880] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064909] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064923] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064938] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064989] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.064993] [] ? strlcpy+0x4a/0x60 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065009] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065023] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065038] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065053] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065097] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065137] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065191] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065245] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065299] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065351] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065354] [] ? thread_return+0x4e/0x7d0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065406] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065410] [] ? kthread+0x9e/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065412] [] ? child_rip+0xa/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065415] [] ? kthread+0x0/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.065417] [] ? child_rip+0x0/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066401] BUG: soft lockup - CPU#7 stuck for 68s! [mdt01_001:21486] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066402] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066442] CPU 7 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066443] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066478] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066480] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066483] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066487] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066488] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066490] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066491] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066493] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066495] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066497] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066498] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066500] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066503] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066505] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066506] Stack: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066507] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066509] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066512] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066514] Call Trace: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066548] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066595] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066645] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066691] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066738] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066754] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066765] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066777] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066787] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066802] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066818] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066847] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066862] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066877] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066928] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066931] [] ? strlcpy+0x4a/0x60 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066948] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066962] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066977] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.066992] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067036] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067076] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067130] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067184] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067238] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067289] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067293] [] ? thread_return+0x4e/0x7d0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067345] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067348] [] ? kthread+0x9e/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067351] [] ? child_rip+0xa/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067354] [] ? kthread+0x0/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067356] [] ? child_rip+0x0/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067357] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067372] Call Trace: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067415] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067461] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067511] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067557] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067604] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067619] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067630] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067642] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067652] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067667] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067683] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067712] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067727] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067741] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067791] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067795] [] ? strlcpy+0x4a/0x60 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067812] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067826] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067841] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067856] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067900] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067940] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.067994] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.068048] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.068102] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.068154] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.068157] [] ? thread_return+0x4e/0x7d0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.068209] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.068212] [] ? kthread+0x9e/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.068215] [] ? child_rip+0xa/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.068218] [] ? kthread+0x0/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483006.068220] [] ? child_rip+0x0/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.573109] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.670257] CPU 4 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.672380] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.769892] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.771757] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.785182] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.793797] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.799656] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.810194] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.818071] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.825976] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.833871] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.841793] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.850624] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.856917] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.864780] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.872675] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.880511] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.889920] Stack: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.892323] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.900148] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.908851] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.917994] Call Trace: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.921207] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.928750] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.936787] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.944404] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.951908] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.959393] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.966946] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.974236] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.981325] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.988220] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483008.996029] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.003165] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.011403] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.017730] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.024759] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.032669] [] ? strlcpy+0x4a/0x60 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.038385] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.045328] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.053022] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.060406] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.067759] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.075338] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.083396] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.090754] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.097722] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.105607] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.112682] [] ? thread_return+0x4e/0x7d0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.118987] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.125900] [] ? kthread+0x9e/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.131455] [] ? child_rip+0xa/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.137140] [] ? kthread+0x0/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.142634] [] ? child_rip+0x0/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.148319] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.169891] Call Trace: Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.172785] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.180181] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.188227] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.195857] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.203253] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.210718] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.218288] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.225630] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.232639] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.239492] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.247257] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.254422] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.262684] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.269038] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.275998] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.283828] [] ? strlcpy+0x4a/0x60 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.289423] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.296269] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.303636] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.310847] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.318217] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.325806] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.333872] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.341324] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.348293] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.356184] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.363297] [] ? thread_return+0x4e/0x7d0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.369646] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.376593] [] ? kthread+0x9e/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.382174] [] ? child_rip+0xa/0x20 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.387841] [] ? kthread+0x0/0xc0 Aug 25 20:12:27 feral28.ccs.ornl.gov kernel: [1483009.393299] [] ? child_rip+0x0/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.063009] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066009] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066011] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066053] CPU 5 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066054] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066090] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066092] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066095] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066099] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066101] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066103] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066104] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066106] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066108] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066110] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066112] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066113] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066115] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066117] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066119] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066120] Stack: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066121] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066123] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066125] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066127] Call Trace: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066171] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066218] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066267] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066314] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066361] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066377] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066388] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066400] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066410] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066426] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066442] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066471] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066486] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066501] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066551] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066555] [] ? strlcpy+0x4a/0x60 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066571] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066586] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066601] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066616] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066660] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066700] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066754] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066808] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066862] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066913] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066917] [] ? thread_return+0x4e/0x7d0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066969] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066972] [] ? kthread+0x9e/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066975] [] ? child_rip+0xa/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066978] [] ? kthread+0x0/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066980] [] ? child_rip+0x0/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066981] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.066996] Call Trace: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067039] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067085] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067135] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067181] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067228] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067243] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067254] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067266] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067276] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067291] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067307] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067336] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067351] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067366] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067416] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067420] [] ? strlcpy+0x4a/0x60 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067436] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067450] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067465] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067480] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067524] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067565] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067618] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067673] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067726] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067778] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067781] [] ? thread_return+0x4e/0x7d0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067833] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067837] [] ? kthread+0x9e/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067839] [] ? child_rip+0xa/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067842] [] ? kthread+0x0/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.067844] [] ? child_rip+0x0/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070010] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070011] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070054] CPU 6 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070055] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070093] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070094] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070098] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070102] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070104] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070105] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070107] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070109] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070110] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070112] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070114] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070116] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070118] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070122] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070123] Stack: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070124] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070126] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070128] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070131] Call Trace: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070166] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070212] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070262] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070309] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070355] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070371] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070382] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070394] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070406] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070456] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070467] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070482] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070514] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070530] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070559] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070574] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070589] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070639] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070643] [] ? strlcpy+0x4a/0x60 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070659] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070674] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070689] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070704] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070747] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070788] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070842] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070896] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.070950] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071001] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071005] [] ? thread_return+0x4e/0x7d0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071057] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071060] [] ? kthread+0x9e/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071063] [] ? child_rip+0xa/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071065] [] ? kthread+0x0/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071068] [] ? child_rip+0x0/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071069] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071084] Call Trace: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071127] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071173] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071223] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071269] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071316] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071331] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071342] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071354] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071366] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071416] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071426] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071442] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071473] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071489] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071518] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071533] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071548] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071598] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071602] [] ? strlcpy+0x4a/0x60 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071618] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071633] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071648] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071663] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071706] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071747] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071801] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071855] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071908] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071960] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.071963] [] ? thread_return+0x4e/0x7d0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.072016] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.072019] [] ? kthread+0x9e/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.072022] [] ? child_rip+0xa/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.072025] [] ? kthread+0x0/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.072027] [] ? child_rip+0x0/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073009] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073010] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073050] CPU 7 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073051] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073087] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073088] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073091] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073095] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073097] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073098] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073100] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073102] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073103] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073105] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073107] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073108] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073110] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073112] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073113] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073115] Stack: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073116] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073118] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073120] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073123] Call Trace: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073157] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073203] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073253] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073299] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073346] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073362] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073373] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073385] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073395] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073410] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073427] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073455] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073470] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073485] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073535] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073539] [] ? strlcpy+0x4a/0x60 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073556] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073571] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073585] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073600] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073644] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073684] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073738] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073791] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073845] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073897] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073900] [] ? thread_return+0x4e/0x7d0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073952] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073955] [] ? kthread+0x9e/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073958] [] ? child_rip+0xa/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073961] [] ? kthread+0x0/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073963] [] ? child_rip+0x0/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073965] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.073980] Call Trace: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074022] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074069] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074119] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074165] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074212] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074227] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074238] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074250] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074260] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074275] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074291] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074320] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074334] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074349] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074400] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074403] [] ? strlcpy+0x4a/0x60 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074420] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074434] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074449] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074464] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074508] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074548] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074602] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074656] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074709] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074761] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074765] [] ? thread_return+0x4e/0x7d0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074816] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074820] [] ? kthread+0x9e/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074822] [] ? child_rip+0xa/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074825] [] ? kthread+0x0/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483090.074827] [] ? child_rip+0x0/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.600657] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.700231] CPU 4 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.702442] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.800128] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.802018] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.815442] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.824153] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.830116] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.838055] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.845984] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.853898] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.861819] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.869750] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.878737] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.885056] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.892977] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.900943] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.908881] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.918326] Stack: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.920755] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.928667] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.937204] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.946114] Call Trace: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.949000] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.956378] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.964441] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.972083] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.979612] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.987124] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483092.994737] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.002061] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.009201] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.016086] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.023924] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.031121] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.039419] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.045755] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.052698] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.060564] [] ? strlcpy+0x4a/0x60 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.066193] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.072995] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.080345] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.087520] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.094838] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.102427] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.110536] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.117999] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.124966] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.132843] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.139962] [] ? thread_return+0x4e/0x7d0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.146260] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.153136] [] ? kthread+0x9e/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.158760] [] ? child_rip+0xa/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.164497] [] ? kthread+0x0/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.169975] [] ? child_rip+0x0/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.175597] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.197075] Call Trace: Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.200012] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.207433] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.215454] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.223086] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.230567] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.238144] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.245776] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.253195] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.260318] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.267229] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.275066] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.282245] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.290501] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.296872] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.303893] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.311811] [] ? strlcpy+0x4a/0x60 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.317457] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.324250] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.331617] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.338855] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.346129] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.353657] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.361800] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.369182] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.376170] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.384020] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.391165] [] ? thread_return+0x4e/0x7d0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.397532] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.404400] [] ? kthread+0x9e/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.410042] [] ? child_rip+0xa/0x20 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.415744] [] ? kthread+0x0/0xc0 Aug 25 20:13:51 feral28.ccs.ornl.gov kernel: [1483093.421273] [] ? child_rip+0x0/0x20 Aug 25 20:14:06 feral28.ccs.ornl.gov kernel: [1483108.395343] Lustre: sultan-MDT000b: haven't heard from client ade025d5-35bd-119e-3200-b4438b686c43 (at 20@gni1) in 336 seconds. I think it's dead, and I am evicting it. exp ffff880ba0d9d000, cur 1472170446 expire 1472170224 last 1472170110 Aug 25 20:14:06 feral28.ccs.ornl.gov kernel: [1483108.418071] Lustre: Skipped 1 previous similar message Aug 25 20:15:01 feral28.ccs.ornl.gov kernel: [1483164.083952] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 20:15:01 feral28.ccs.ornl.gov kernel: [1483164.094989] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 32 previous similar messages Aug 25 20:15:01 feral28.ccs.ornl.gov kernel: [1483164.106061] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.202.61@o2ib1 (223): c: 63, oc: 0, rc: 63 Aug 25 20:15:01 feral28.ccs.ornl.gov kernel: [1483164.119448] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 32 previous similar messages Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.069617] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072617] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072619] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072662] CPU 5 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072663] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072700] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072701] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072704] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072709] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072711] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072712] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072714] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072716] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072717] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072719] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072721] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072723] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072725] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072726] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072728] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072729] Stack: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072730] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072732] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072734] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072737] Call Trace: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072780] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072827] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072877] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072924] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072970] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072986] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.072997] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073009] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073019] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073034] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073050] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073080] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073095] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073109] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073160] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073164] [] ? strlcpy+0x4a/0x60 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073180] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073195] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073210] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073225] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073269] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073309] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073363] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073418] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073471] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073523] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073527] [] ? thread_return+0x4e/0x7d0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073579] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073582] [] ? kthread+0x9e/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073585] [] ? child_rip+0xa/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073588] [] ? kthread+0x0/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073590] [] ? child_rip+0x0/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073591] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073606] Call Trace: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073649] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073696] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073745] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073792] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073838] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073854] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073864] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073876] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073887] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073902] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073918] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073947] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073961] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.073976] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074026] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074030] [] ? strlcpy+0x4a/0x60 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074046] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074061] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074076] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074091] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074134] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074175] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074229] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074283] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074337] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074389] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074392] [] ? thread_return+0x4e/0x7d0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074444] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074447] [] ? kthread+0x9e/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074450] [] ? child_rip+0xa/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074453] [] ? kthread+0x0/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.074455] [] ? child_rip+0x0/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076618] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076619] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076663] CPU 6 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076664] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076700] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076702] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076705] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076710] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076711] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076713] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076715] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076716] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076718] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076720] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076722] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076724] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076726] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076728] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076729] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076731] Stack: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076732] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076734] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076736] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076738] Call Trace: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076773] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076820] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076870] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076916] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076963] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076979] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.076990] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077001] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077014] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077064] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077074] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077090] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077121] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077137] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077166] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077181] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077196] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077246] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077251] [] ? strlcpy+0x4a/0x60 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077267] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077282] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077296] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077311] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077355] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077395] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077450] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077503] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077557] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077609] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077612] [] ? thread_return+0x4e/0x7d0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077664] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077667] [] ? kthread+0x9e/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077670] [] ? child_rip+0xa/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077673] [] ? kthread+0x0/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077675] [] ? child_rip+0x0/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077676] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077692] Call Trace: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077734] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077781] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077830] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077877] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077923] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077939] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077949] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077961] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.077973] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078023] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078034] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078049] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078078] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078095] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078124] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078138] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078153] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078204] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078208] [] ? strlcpy+0x4a/0x60 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078224] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078239] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078254] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078268] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078312] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078352] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078406] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078460] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078513] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078565] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078569] [] ? thread_return+0x4e/0x7d0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078620] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078624] [] ? kthread+0x9e/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078627] [] ? child_rip+0xa/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078629] [] ? kthread+0x0/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.078632] [] ? child_rip+0x0/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079616] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079618] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079657] CPU 7 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079658] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079693] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079695] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079698] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079702] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079704] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079705] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079707] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079708] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079710] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079712] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079714] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079715] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079717] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079718] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079720] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079722] Stack: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079723] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079725] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079727] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079730] Call Trace: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079763] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079810] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079860] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079907] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079953] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079969] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079979] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.079991] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080002] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080017] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080033] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080062] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080077] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080092] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080142] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080146] [] ? strlcpy+0x4a/0x60 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080162] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080177] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080192] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080207] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080250] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080291] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080345] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080399] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080452] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080504] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080507] [] ? thread_return+0x4e/0x7d0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080559] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080562] [] ? kthread+0x9e/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080565] [] ? child_rip+0xa/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080568] [] ? kthread+0x0/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080570] [] ? child_rip+0x0/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080571] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080586] Call Trace: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080629] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080675] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080725] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080771] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080818] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080833] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080844] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080856] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080867] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080882] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080898] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080926] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080941] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.080956] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081006] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081010] [] ? strlcpy+0x4a/0x60 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081026] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081041] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081055] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081070] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081114] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081154] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081208] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081262] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081316] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081367] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081371] [] ? thread_return+0x4e/0x7d0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081422] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081426] [] ? kthread+0x9e/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081428] [] ? child_rip+0xa/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081431] [] ? kthread+0x0/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483174.081433] [] ? child_rip+0x0/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.591050] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.688094] CPU 4 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.690183] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.787565] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.789438] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.802888] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.811477] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.817300] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.825180] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.833093] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.841021] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.848934] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.856813] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.865663] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.871981] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.879808] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.887719] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.895600] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.905045] Stack: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.907436] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.915349] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.923818] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.932624] Call Trace: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.935516] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.942955] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.950995] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.958601] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.965994] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.973512] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.981082] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.988398] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483176.995434] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.002312] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.010104] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.017189] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.025469] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.031805] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.038809] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.046700] [] ? strlcpy+0x4a/0x60 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.052347] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.059208] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.066603] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.073831] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.081123] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.088729] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.096821] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.104274] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.111302] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.119240] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.126298] [] ? thread_return+0x4e/0x7d0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.132587] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.139558] [] ? kthread+0x9e/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.145104] [] ? child_rip+0xa/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.150825] [] ? kthread+0x0/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.156354] [] ? child_rip+0x0/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.162073] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.183620] Call Trace: Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.186521] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.193952] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.201998] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.209587] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.217058] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.224611] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.232198] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.239504] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.246524] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.253409] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.261210] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.268393] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.276630] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.282948] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.289976] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.300505] [] ? strlcpy+0x4a/0x60 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.306140] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.312988] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.320389] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.327618] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.334953] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.342550] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.350694] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.358086] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.365072] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.372965] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.380032] [] ? thread_return+0x4e/0x7d0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.386330] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.393241] [] ? kthread+0x9e/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.398847] [] ? child_rip+0xa/0x20 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.404750] [] ? kthread+0x0/0xc0 Aug 25 20:15:15 feral28.ccs.ornl.gov kernel: [1483177.410288] [] ? child_rip+0x0/0x20 Aug 25 20:16:26 feral28.ccs.ornl.gov kernel: [1483248.305008] Lustre: sultan-MDT000b: haven't heard from client b8e09222-09d9-8a4a-26f0-d58195f742d1 (at 18@gni1) in 337 seconds. I think it's dead, and I am evicting it. exp ffff880b97597400, cur 1472170586 expire 1472170364 last 1472170249 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.076225] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079226] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079227] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079269] CPU 5 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079270] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079306] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079308] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079311] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079315] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079317] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079319] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079321] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079322] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079324] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079326] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079328] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079330] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079331] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079333] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079335] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079336] Stack: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079337] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079339] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079341] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079343] Call Trace: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079387] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079433] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079483] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079530] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079577] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079593] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079603] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079616] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079627] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079642] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079658] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079687] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079702] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079717] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079767] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079771] [] ? strlcpy+0x4a/0x60 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079787] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079802] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079816] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079831] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079875] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079915] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.079969] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080024] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080077] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080129] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080132] [] ? thread_return+0x4e/0x7d0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080184] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080188] [] ? kthread+0x9e/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080191] [] ? child_rip+0xa/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080194] [] ? kthread+0x0/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080196] [] ? child_rip+0x0/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080197] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080212] Call Trace: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080255] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080301] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080351] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080397] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080444] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080460] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080470] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080482] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080492] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080507] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080524] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080552] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080567] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080582] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080632] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080636] [] ? strlcpy+0x4a/0x60 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080652] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080667] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080681] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080696] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080740] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080781] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080834] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080889] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080942] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080994] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.080997] [] ? thread_return+0x4e/0x7d0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.081049] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.081053] [] ? kthread+0x9e/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.081055] [] ? child_rip+0xa/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.081058] [] ? kthread+0x0/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.081060] [] ? child_rip+0x0/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083226] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083227] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083270] CPU 6 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083271] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083308] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083310] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083312] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083317] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083319] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083320] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083322] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083323] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083325] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083327] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083329] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083331] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083333] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083334] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083336] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083338] Stack: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083339] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083341] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083343] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083346] Call Trace: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083381] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083428] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083478] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083524] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083571] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083587] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083597] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083609] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083622] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083672] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083682] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083698] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083729] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083746] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083774] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083789] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083804] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083854] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083859] [] ? strlcpy+0x4a/0x60 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083875] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083890] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083904] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083919] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.083963] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084003] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084058] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084112] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084165] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084217] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084221] [] ? thread_return+0x4e/0x7d0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084272] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084276] [] ? kthread+0x9e/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084279] [] ? child_rip+0xa/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084282] [] ? kthread+0x0/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084284] [] ? child_rip+0x0/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084285] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084301] Call Trace: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084343] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084390] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084439] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084486] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084533] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084548] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084559] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084570] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084583] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084633] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084644] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084659] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084690] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084706] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084735] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084749] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084765] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084815] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084819] [] ? strlcpy+0x4a/0x60 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084835] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084849] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084864] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084879] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084923] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.084963] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085017] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085071] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085124] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085176] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085180] [] ? thread_return+0x4e/0x7d0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085232] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085235] [] ? kthread+0x9e/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085238] [] ? child_rip+0xa/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085241] [] ? kthread+0x0/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.085243] [] ? child_rip+0x0/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086225] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086226] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086267] CPU 7 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086268] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086305] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086307] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086309] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086313] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086315] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086316] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086318] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086320] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086321] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086323] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086325] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086327] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086329] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086330] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086332] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086334] Stack: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086335] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086337] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086339] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086341] Call Trace: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086376] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086423] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086472] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086519] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086566] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086581] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086592] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086604] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086615] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086630] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086646] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086675] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086690] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086705] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086755] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086759] [] ? strlcpy+0x4a/0x60 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086775] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086790] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086805] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086820] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086863] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086903] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.086958] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087011] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087065] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087117] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087120] [] ? thread_return+0x4e/0x7d0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087172] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087175] [] ? kthread+0x9e/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087178] [] ? child_rip+0xa/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087181] [] ? kthread+0x0/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087183] [] ? child_rip+0x0/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087184] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087200] Call Trace: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087242] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087289] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087338] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087385] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087431] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087447] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087457] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087469] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087480] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087495] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087511] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087540] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087554] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087569] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087619] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087623] [] ? strlcpy+0x4a/0x60 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087639] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087654] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087669] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087684] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087728] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087768] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087822] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087876] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087929] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087981] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.087984] [] ? thread_return+0x4e/0x7d0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.088036] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.088039] [] ? kthread+0x9e/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.088042] [] ? child_rip+0xa/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.088045] [] ? kthread+0x0/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483258.088047] [] ? child_rip+0x0/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.615114] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.713473] CPU 4 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.715616] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.813561] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.815435] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.828849] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.837578] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.843453] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.851444] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.859323] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.867225] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.875158] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.883131] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.892110] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.898507] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.906445] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.914393] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.922306] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.931787] Stack: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.934223] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.942179] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.950759] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.959617] Call Trace: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.962511] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.969959] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.977987] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.985752] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483260.993368] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.001112] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.008870] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.016353] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.023563] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.030664] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.038739] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.046123] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.054630] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.064319] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.071503] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.079562] [] ? strlcpy+0x4a/0x60 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.085217] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.092070] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.099490] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.106709] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.114002] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.121564] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.129664] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.137115] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.144111] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.151962] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.159047] [] ? thread_return+0x4e/0x7d0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.165361] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.172264] [] ? kthread+0x9e/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.177931] [] ? child_rip+0xa/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.183616] [] ? kthread+0x0/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.189198] [] ? child_rip+0x0/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.194856] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.216324] Call Trace: Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.219244] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.226614] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.234669] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.242343] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.249858] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.257409] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.265068] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.272437] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.279559] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.286480] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.294307] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.301436] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.309689] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.316121] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.323141] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.331077] [] ? strlcpy+0x4a/0x60 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.336678] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.343482] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.350884] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.358111] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.365413] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.373053] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.381179] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.388615] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.395601] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.403468] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.410510] [] ? thread_return+0x4e/0x7d0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.416782] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.423751] [] ? kthread+0x9e/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.429428] [] ? child_rip+0xa/0x20 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.435149] [] ? kthread+0x0/0xc0 Aug 25 20:16:39 feral28.ccs.ornl.gov kernel: [1483261.440687] [] ? child_rip+0x0/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.082833] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085834] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085835] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085878] CPU 5 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085879] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085915] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085917] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085920] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085924] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085926] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085927] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085929] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085931] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085932] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085934] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085936] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085938] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085940] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085941] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085943] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085945] Stack: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085945] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085947] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085949] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085952] Call Trace: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.085996] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086042] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086092] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086139] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086186] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086201] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086212] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086224] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086235] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086250] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086266] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086296] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086310] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086325] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086376] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086380] [] ? strlcpy+0x4a/0x60 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086396] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086410] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086425] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086440] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086484] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086524] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086578] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086632] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086686] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086738] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086742] [] ? thread_return+0x4e/0x7d0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086794] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086797] [] ? kthread+0x9e/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086800] [] ? child_rip+0xa/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086803] [] ? kthread+0x0/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086805] [] ? child_rip+0x0/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086806] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086821] Call Trace: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086864] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086910] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.086960] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087006] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087053] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087068] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087079] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087091] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087101] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087117] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087132] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087161] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087176] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087191] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087241] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087245] [] ? strlcpy+0x4a/0x60 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087261] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087276] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087291] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087306] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087349] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087390] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087444] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087498] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087552] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087603] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087607] [] ? thread_return+0x4e/0x7d0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087659] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087662] [] ? kthread+0x9e/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087665] [] ? child_rip+0xa/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087668] [] ? kthread+0x0/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.087670] [] ? child_rip+0x0/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089834] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089836] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089879] CPU 6 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089879] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089916] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089918] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089921] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089926] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089927] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089929] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089931] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089932] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089934] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089936] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089938] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089940] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089942] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089943] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089945] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089947] Stack: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089948] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089950] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089952] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089955] Call Trace: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.089990] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090036] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090086] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090133] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090179] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090195] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090206] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090217] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090230] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090280] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090290] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090305] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090337] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090353] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090382] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090397] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090412] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090462] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090466] [] ? strlcpy+0x4a/0x60 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090482] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090497] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090512] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090527] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090571] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090611] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090665] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090719] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090773] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090825] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090828] [] ? thread_return+0x4e/0x7d0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090880] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090883] [] ? kthread+0x9e/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090886] [] ? child_rip+0xa/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090889] [] ? kthread+0x0/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090891] [] ? child_rip+0x0/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090893] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090908] Call Trace: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090950] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.090997] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091046] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091093] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091140] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091155] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091166] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091177] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091189] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091239] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091250] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091265] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091294] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091311] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091339] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091354] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091369] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091419] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091423] [] ? strlcpy+0x4a/0x60 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091439] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091454] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091469] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091483] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091527] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091567] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091621] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091675] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091728] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091780] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091784] [] ? thread_return+0x4e/0x7d0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091835] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091839] [] ? kthread+0x9e/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091842] [] ? child_rip+0xa/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091844] [] ? kthread+0x0/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.091846] [] ? child_rip+0x0/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092833] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092834] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092875] CPU 7 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092876] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092911] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092913] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092916] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092920] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092922] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092923] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092925] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092926] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092928] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092930] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092932] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092933] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092937] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092938] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092940] Stack: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092940] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092943] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092945] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092947] Call Trace: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.092981] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093028] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093077] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093124] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093171] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093186] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093197] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093209] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093220] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093235] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093251] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093280] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093294] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093309] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093360] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093363] [] ? strlcpy+0x4a/0x60 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093380] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093394] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093409] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093424] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093468] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093508] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093562] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093616] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093670] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093722] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093725] [] ? thread_return+0x4e/0x7d0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093777] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093780] [] ? kthread+0x9e/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093783] [] ? child_rip+0xa/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093786] [] ? kthread+0x0/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093788] [] ? child_rip+0x0/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093789] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093804] Call Trace: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093847] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093893] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093943] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.093990] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094036] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094052] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094062] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094074] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094085] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094100] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094116] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094145] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094159] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094174] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094224] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094228] [] ? strlcpy+0x4a/0x60 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094244] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094259] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094274] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094288] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094332] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094372] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094426] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094480] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094534] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094585] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094589] [] ? thread_return+0x4e/0x7d0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094640] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094644] [] ? kthread+0x9e/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094647] [] ? child_rip+0xa/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094649] [] ? kthread+0x0/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483342.094651] [] ? child_rip+0x0/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.598711] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.695643] CPU 4 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.697748] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.795009] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.796873] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.810263] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.818965] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.824797] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.832710] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.840555] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.851154] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.859031] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.866920] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.875795] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.882138] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.890026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.897902] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.905792] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.915211] Stack: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.917630] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.925525] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.934062] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.942834] Call Trace: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.945769] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.953131] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.961142] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.968775] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.976186] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.983695] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.991308] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483344.998668] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.005713] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.012564] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.020331] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.027468] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.035748] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.042075] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.049088] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.056997] [] ? strlcpy+0x4a/0x60 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.062626] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.069461] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.076838] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.083979] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.091237] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.098817] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.106944] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.114370] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.121321] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.129172] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.136213] [] ? thread_return+0x4e/0x7d0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.142493] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.149370] [] ? kthread+0x9e/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.154923] [] ? child_rip+0xa/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.160584] [] ? kthread+0x0/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.166085] [] ? child_rip+0x0/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.171701] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.193187] Call Trace: Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.196071] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.203442] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.211446] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.219095] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.226582] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.234127] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.241714] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.249057] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.256145] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.262988] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.270796] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.277961] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.286163] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.292507] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.299501] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.307394] [] ? strlcpy+0x4a/0x60 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.312989] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.319825] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.327201] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.334369] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.341713] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.349284] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.357306] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.364725] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.371710] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.379595] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.386747] [] ? thread_return+0x4e/0x7d0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.393029] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.399984] [] ? kthread+0x9e/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.405554] [] ? child_rip+0xa/0x20 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.411207] [] ? kthread+0x0/0xc0 Aug 25 20:18:03 feral28.ccs.ornl.gov kernel: [1483345.416699] [] ? child_rip+0x0/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.089442] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092442] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092443] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092486] CPU 5 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092486] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092522] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092524] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092527] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092531] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092532] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092534] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092536] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092537] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092539] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092541] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092543] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092545] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092548] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092550] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092552] Stack: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092552] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092555] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092557] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092559] Call Trace: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092603] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092650] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092699] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092746] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092793] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092808] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092819] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092831] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092841] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092857] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092873] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092902] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092917] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092932] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092982] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.092987] [] ? strlcpy+0x4a/0x60 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093003] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093018] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093032] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093047] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093091] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093131] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093186] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093240] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093293] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093345] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093349] [] ? thread_return+0x4e/0x7d0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093401] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093404] [] ? kthread+0x9e/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093407] [] ? child_rip+0xa/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093409] [] ? kthread+0x0/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093412] [] ? child_rip+0x0/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093413] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093428] Call Trace: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093470] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093517] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093567] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093613] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093660] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093676] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093686] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093698] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093709] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093724] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093740] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093768] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093783] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093798] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093848] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093852] [] ? strlcpy+0x4a/0x60 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093868] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093883] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093898] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093913] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093956] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.093997] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094051] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094105] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094159] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094210] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094214] [] ? thread_return+0x4e/0x7d0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094266] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094269] [] ? kthread+0x9e/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094272] [] ? child_rip+0xa/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094275] [] ? kthread+0x0/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.094277] [] ? child_rip+0x0/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096442] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096444] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096486] CPU 6 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096487] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096524] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096525] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096529] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096533] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096535] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096537] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096538] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096540] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096542] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096544] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096546] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096548] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096551] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096553] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096554] Stack: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096555] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096557] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096559] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096561] Call Trace: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096596] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096643] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096693] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096740] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096786] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096802] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096813] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096825] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096837] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096887] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096898] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096913] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096944] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096961] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.096989] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097004] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097019] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097069] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097073] [] ? strlcpy+0x4a/0x60 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097090] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097104] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097119] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097134] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097178] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097218] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097273] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097327] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097380] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097432] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097436] [] ? thread_return+0x4e/0x7d0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097487] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097491] [] ? kthread+0x9e/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097494] [] ? child_rip+0xa/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097497] [] ? kthread+0x0/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097499] [] ? child_rip+0x0/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097500] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097515] Call Trace: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097558] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097604] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097654] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097700] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097747] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097762] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097773] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097785] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097797] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097847] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097857] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097873] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097902] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097919] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097947] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097962] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.097977] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098027] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098031] [] ? strlcpy+0x4a/0x60 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098047] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098062] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098077] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098092] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098136] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098176] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098230] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098284] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098338] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098390] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098393] [] ? thread_return+0x4e/0x7d0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098445] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098448] [] ? kthread+0x9e/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098451] [] ? child_rip+0xa/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098454] [] ? kthread+0x0/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.098456] [] ? child_rip+0x0/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099441] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099442] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099482] CPU 7 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099483] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099519] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099521] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099524] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099528] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099529] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099531] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099532] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099534] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099535] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099537] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099539] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099541] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099544] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099546] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099547] Stack: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099548] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099550] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099552] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099554] Call Trace: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099588] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099635] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099684] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099731] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099777] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099793] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099804] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099815] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099826] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099841] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099857] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099886] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099901] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099916] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099966] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099970] [] ? strlcpy+0x4a/0x60 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.099986] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100001] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100015] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100030] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100074] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100114] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100168] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100222] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100276] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100328] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100331] [] ? thread_return+0x4e/0x7d0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100383] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100386] [] ? kthread+0x9e/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100389] [] ? child_rip+0xa/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100391] [] ? kthread+0x0/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100394] [] ? child_rip+0x0/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100395] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100410] Call Trace: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100452] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100499] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100548] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100595] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100642] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100657] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100668] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100680] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100690] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100705] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100721] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100750] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100765] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100779] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100830] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100834] [] ? strlcpy+0x4a/0x60 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100850] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100864] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100879] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100894] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100938] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.100978] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101032] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101086] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101139] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101191] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101195] [] ? thread_return+0x4e/0x7d0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101246] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101250] [] ? kthread+0x9e/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101253] [] ? child_rip+0xa/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101256] [] ? kthread+0x0/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483426.101258] [] ? child_rip+0x0/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.613310] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.710207] CPU 4 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.712294] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.809834] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.811689] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.825019] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.833606] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.839456] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.847397] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.855257] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.863119] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.871013] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.878891] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.887777] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.894095] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.901999] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.909869] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.917746] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.927176] Stack: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.929586] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.937499] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.946001] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.954805] Call Trace: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.957709] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.965164] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.973220] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.980860] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.988324] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483428.995877] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.003429] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.010745] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.017790] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.024590] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.032400] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.039552] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.047860] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.054179] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.061174] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.069064] [] ? strlcpy+0x4a/0x60 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.074695] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.081480] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.088820] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.096029] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.103306] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.110894] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.118995] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.126465] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.133494] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.141370] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.148532] [] ? thread_return+0x4e/0x7d0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.154840] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.161715] [] ? kthread+0x9e/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.167296] [] ? child_rip+0xa/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.173008] [] ? kthread+0x0/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.178518] [] ? child_rip+0x0/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.184230] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.205758] Call Trace: Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.208652] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.216031] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.224017] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.231665] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.239094] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.246603] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.254207] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.261550] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.268630] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.275499] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.283351] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.290513] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.298760] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.305036] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.312020] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.319932] [] ? strlcpy+0x4a/0x60 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.325629] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.332423] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.339842] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.347042] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.354300] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.361908] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.369964] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.377390] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.384308] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.392253] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.399320] [] ? thread_return+0x4e/0x7d0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.405713] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.412625] [] ? kthread+0x9e/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.418152] [] ? child_rip+0xa/0x20 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.423804] [] ? kthread+0x0/0xc0 Aug 25 20:19:27 feral28.ccs.ornl.gov kernel: [1483429.429299] [] ? child_rip+0x0/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.096050] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099050] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099052] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099094] CPU 5 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099095] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099131] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099133] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099135] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099140] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099141] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099143] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099144] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099146] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099147] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099149] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099151] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099153] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099155] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099156] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099158] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099160] Stack: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099161] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099163] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099165] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099167] Call Trace: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099212] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099258] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099308] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099355] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099402] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099417] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099428] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099440] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099451] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099466] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099482] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099511] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099526] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099541] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099591] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099595] [] ? strlcpy+0x4a/0x60 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099612] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099626] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099641] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099656] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099700] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099740] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099794] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099849] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099902] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099954] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.099958] [] ? thread_return+0x4e/0x7d0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100009] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100013] [] ? kthread+0x9e/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100016] [] ? child_rip+0xa/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100018] [] ? kthread+0x0/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100021] [] ? child_rip+0x0/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100022] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100037] Call Trace: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100080] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100126] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100176] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100222] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100269] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100284] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100295] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100307] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100317] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100332] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100349] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100377] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100392] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100407] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100457] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100461] [] ? strlcpy+0x4a/0x60 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100477] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100491] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100506] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100521] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100565] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100606] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100660] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100714] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100767] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100819] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100823] [] ? thread_return+0x4e/0x7d0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100875] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100878] [] ? kthread+0x9e/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100881] [] ? child_rip+0xa/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100884] [] ? kthread+0x0/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.100886] [] ? child_rip+0x0/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103051] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103052] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103096] CPU 6 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103097] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103132] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103134] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103137] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103142] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103143] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103145] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103147] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103148] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103150] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103152] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103154] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103156] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103158] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103160] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103161] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103163] Stack: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103164] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103166] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103168] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103170] Call Trace: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103205] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103251] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103301] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103348] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103394] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103410] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103421] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103433] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103445] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103495] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103506] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103521] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103552] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103569] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103597] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103612] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103627] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103677] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103681] [] ? strlcpy+0x4a/0x60 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103698] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103712] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103727] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103742] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103786] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103826] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103881] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103935] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.103988] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104040] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104043] [] ? thread_return+0x4e/0x7d0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104095] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104099] [] ? kthread+0x9e/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104102] [] ? child_rip+0xa/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104105] [] ? kthread+0x0/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104107] [] ? child_rip+0x0/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104108] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104123] Call Trace: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104166] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104213] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104262] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104309] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104355] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104371] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104381] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104393] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104405] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104455] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104466] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104481] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104511] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104527] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104555] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104570] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104585] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104636] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104640] [] ? strlcpy+0x4a/0x60 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104656] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104670] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104685] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104700] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104744] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104784] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104838] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104893] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104946] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.104998] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.105001] [] ? thread_return+0x4e/0x7d0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.105053] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.105056] [] ? kthread+0x9e/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.105059] [] ? child_rip+0xa/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.105062] [] ? kthread+0x0/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.105064] [] ? child_rip+0x0/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106049] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106051] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106090] CPU 7 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106091] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106128] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106129] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106132] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106136] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106138] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106139] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106141] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106142] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106144] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106146] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106148] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106150] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106153] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106155] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106157] Stack: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106158] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106160] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106162] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106164] Call Trace: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106198] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106245] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106295] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106341] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106388] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106403] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106414] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106426] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106437] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106452] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106468] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106497] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106512] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106527] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106577] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106581] [] ? strlcpy+0x4a/0x60 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106597] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106612] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106627] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106642] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106686] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106726] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106780] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106834] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106887] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106939] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106943] [] ? thread_return+0x4e/0x7d0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106994] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.106998] [] ? kthread+0x9e/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107001] [] ? child_rip+0xa/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107003] [] ? kthread+0x0/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107006] [] ? child_rip+0x0/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107007] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107022] Call Trace: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107065] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107111] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107161] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107207] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107254] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107269] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107280] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107292] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107302] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107317] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107334] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107362] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107377] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107392] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107442] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107446] [] ? strlcpy+0x4a/0x60 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107462] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107477] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107492] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107507] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107550] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107591] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107645] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107699] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107753] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107804] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107808] [] ? thread_return+0x4e/0x7d0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107859] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107863] [] ? kthread+0x9e/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107866] [] ? child_rip+0xa/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107869] [] ? kthread+0x0/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483510.107871] [] ? child_rip+0x0/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.633907] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.731270] CPU 4 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.733378] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.831428] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.833300] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.846873] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.855574] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.861528] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.869510] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.877484] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.885431] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.893336] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.901319] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.910201] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.916607] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.924737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.932719] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.940797] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.950269] Stack: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.952670] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.960618] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.969170] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.978186] Call Trace: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.981157] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.988700] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483512.996815] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.004527] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.012005] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.019653] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.027527] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.035189] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.042423] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.049622] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.057728] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.064943] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.073336] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.079887] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.087021] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.095052] [] ? strlcpy+0x4a/0x60 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.100709] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.107623] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.114981] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.122165] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.129545] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.137175] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.145216] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.152687] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.159716] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.167583] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.174685] [] ? thread_return+0x4e/0x7d0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.181035] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.187929] [] ? kthread+0x9e/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.193569] [] ? child_rip+0xa/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.199308] [] ? kthread+0x0/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.204820] [] ? child_rip+0x0/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.210521] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.232111] Call Trace: Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.235011] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.242374] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.250394] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.257931] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.265426] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.272912] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.280541] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.287936] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.295016] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.301883] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.309632] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.316779] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.325033] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.331301] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.338329] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.346221] [] ? strlcpy+0x4a/0x60 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.351852] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.358636] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.366037] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.373222] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.380497] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.388129] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.396212] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.403570] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.410503] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.418432] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.425499] [] ? thread_return+0x4e/0x7d0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.431840] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.441394] [] ? kthread+0x9e/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.446950] [] ? child_rip+0xa/0x20 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.452601] [] ? kthread+0x0/0xc0 Aug 25 20:20:51 feral28.ccs.ornl.gov kernel: [1483513.458058] [] ? child_rip+0x0/0x20 Aug 25 20:21:03 feral28.ccs.ornl.gov kernel: [1483525.419374] Lustre: sultan-MDT000b: Client 8d48f2bb-3a94-5dec-2c5c-1f0ea8b2f1ab (at 25@gni1) reconnecting Aug 25 20:21:03 feral28.ccs.ornl.gov kernel: [1483525.429793] Lustre: Skipped 9 previous similar messages Aug 25 20:21:03 feral28.ccs.ornl.gov kernel: [1483525.435563] Lustre: sultan-MDT000b: Connection restored to 3f7fe8df-22b7-3336-a8c1-f8bfea0cd621 (at 25@gni1) Aug 25 20:21:03 feral28.ccs.ornl.gov kernel: [1483525.446154] Lustre: Skipped 12 previous similar messages Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.102658] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105658] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105660] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105702] CPU 5 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105703] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105740] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105742] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105745] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105749] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105750] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105752] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105754] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105755] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105757] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105759] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105761] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105762] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105764] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105766] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105768] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105769] Stack: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105770] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105772] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105775] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105777] Call Trace: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105821] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105867] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105917] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.105964] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106011] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106026] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106037] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106049] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106060] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106075] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106091] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106120] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106135] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106150] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106200] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106204] [] ? strlcpy+0x4a/0x60 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106220] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106235] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106250] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106265] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106309] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106349] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106403] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106457] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106511] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106563] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106566] [] ? thread_return+0x4e/0x7d0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106618] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106621] [] ? kthread+0x9e/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106624] [] ? child_rip+0xa/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106627] [] ? kthread+0x0/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106629] [] ? child_rip+0x0/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106630] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106645] Call Trace: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106688] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106734] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106784] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106830] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106877] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106893] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106903] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106915] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106925] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106940] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106957] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.106985] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107000] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107015] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107065] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107069] [] ? strlcpy+0x4a/0x60 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107085] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107100] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107115] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107129] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107173] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107214] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107267] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107321] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107375] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107427] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107430] [] ? thread_return+0x4e/0x7d0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107482] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107485] [] ? kthread+0x9e/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107488] [] ? child_rip+0xa/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107491] [] ? kthread+0x0/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.107493] [] ? child_rip+0x0/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109659] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109660] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109703] CPU 6 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109704] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109741] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109743] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109746] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109751] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109753] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109754] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109756] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109758] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109759] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109761] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109763] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109765] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109767] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109771] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109772] Stack: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109773] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109775] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109777] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109779] Call Trace: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109814] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109860] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109910] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.109957] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110004] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110019] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110030] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110042] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110054] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110104] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110115] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110131] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110162] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110178] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110207] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110222] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110237] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110287] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110291] [] ? strlcpy+0x4a/0x60 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110308] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110322] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110337] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110352] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110396] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110436] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110490] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110544] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110598] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110650] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110653] [] ? thread_return+0x4e/0x7d0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110705] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110708] [] ? kthread+0x9e/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110711] [] ? child_rip+0xa/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110714] [] ? kthread+0x0/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110716] [] ? child_rip+0x0/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110717] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110733] Call Trace: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110775] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110822] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110871] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110918] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110965] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110980] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.110991] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111002] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111014] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111065] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111075] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111090] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111119] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111136] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111164] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111179] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111194] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111244] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111248] [] ? strlcpy+0x4a/0x60 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111264] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111279] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111294] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111309] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111353] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111393] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111447] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111501] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111555] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111607] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111610] [] ? thread_return+0x4e/0x7d0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111662] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111665] [] ? kthread+0x9e/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111668] [] ? child_rip+0xa/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111671] [] ? kthread+0x0/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.111673] [] ? child_rip+0x0/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112658] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112659] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112699] CPU 7 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112700] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112736] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112738] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112741] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112745] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112746] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112748] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112750] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112751] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112753] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112755] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112756] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112758] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112760] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112761] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112763] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112765] Stack: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112765] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112767] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112769] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112772] Call Trace: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112806] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112852] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112902] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112948] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.112995] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113010] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113021] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113033] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113044] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113059] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113075] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113104] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113118] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113133] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113184] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113188] [] ? strlcpy+0x4a/0x60 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113204] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113219] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113234] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113248] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113292] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113333] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113387] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113440] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113494] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113546] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113549] [] ? thread_return+0x4e/0x7d0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113601] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113605] [] ? kthread+0x9e/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113607] [] ? child_rip+0xa/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113610] [] ? kthread+0x0/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113612] [] ? child_rip+0x0/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113614] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113629] Call Trace: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113671] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113718] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113768] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113814] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113861] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113876] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113887] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113899] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113909] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113924] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113940] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113969] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113984] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.113999] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114049] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114053] [] ? strlcpy+0x4a/0x60 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114069] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114083] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114098] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114113] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114157] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114197] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114251] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114305] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114359] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114410] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114414] [] ? thread_return+0x4e/0x7d0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114466] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114469] [] ? kthread+0x9e/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114472] [] ? child_rip+0xa/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114475] [] ? kthread+0x0/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483594.114477] [] ? child_rip+0x0/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.636208] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.733727] CPU 4 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.735843] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.833443] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.835332] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.848834] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.857570] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.863439] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.871413] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.879334] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.887256] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.895186] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.903074] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.911975] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.918275] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.926195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.934221] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.942083] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.951590] Stack: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.953991] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.962000] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.970494] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.979342] Call Trace: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.982243] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.989710] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483596.997782] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.005467] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.012972] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.020560] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.028190] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.035576] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.042664] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.049559] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.057438] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.064584] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.072959] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.079278] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.086334] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.094243] [] ? strlcpy+0x4a/0x60 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.099889] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.106673] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.114006] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.121234] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.128553] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.136184] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.144300] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.151790] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.158749] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.169234] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.176328] [] ? thread_return+0x4e/0x7d0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.182633] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.189476] [] ? kthread+0x9e/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.195144] [] ? child_rip+0xa/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.200889] [] ? kthread+0x0/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.206409] [] ? child_rip+0x0/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.212128] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.233580] Call Trace: Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.236473] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.243825] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.251923] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.259677] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.267225] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.274823] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.282445] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.289812] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.296893] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.303804] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.311650] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.318821] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.327057] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.333393] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.340414] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.348322] [] ? strlcpy+0x4a/0x60 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.354022] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.360841] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.368174] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.375385] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.382728] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.390282] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.398391] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.405819] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.412787] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.420706] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.427817] [] ? thread_return+0x4e/0x7d0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.434080] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.441050] [] ? kthread+0x9e/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.446667] [] ? child_rip+0xa/0x20 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.452405] [] ? kthread+0x0/0xc0 Aug 25 20:22:15 feral28.ccs.ornl.gov kernel: [1483597.457934] [] ? child_rip+0x0/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.109267] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112267] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112268] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112311] CPU 5 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112312] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112347] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112349] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112352] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112356] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112358] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112359] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112361] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112362] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112364] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112366] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112368] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112369] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112373] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112375] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112376] Stack: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112377] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112379] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112382] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112384] Call Trace: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112428] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112474] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112524] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112571] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112617] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112633] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112644] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112656] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112667] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112682] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112698] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112727] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112742] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112757] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112807] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112811] [] ? strlcpy+0x4a/0x60 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112827] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112842] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112857] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112872] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112916] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.112956] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113010] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113064] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113118] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113170] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113173] [] ? thread_return+0x4e/0x7d0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113225] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113228] [] ? kthread+0x9e/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113231] [] ? child_rip+0xa/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113234] [] ? kthread+0x0/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113236] [] ? child_rip+0x0/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113237] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113252] Call Trace: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113295] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113341] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113391] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113437] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113484] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113499] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113510] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113522] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113532] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113547] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113563] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113592] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113607] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113622] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113672] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113676] [] ? strlcpy+0x4a/0x60 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113692] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113707] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113722] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113736] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113780] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113821] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113875] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113929] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.113982] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.114034] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.114038] [] ? thread_return+0x4e/0x7d0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.114089] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.114093] [] ? kthread+0x9e/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.114096] [] ? child_rip+0xa/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.114098] [] ? kthread+0x0/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.114101] [] ? child_rip+0x0/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116267] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116269] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116312] CPU 6 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116313] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116348] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116350] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116353] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116358] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116359] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116361] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116363] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116364] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116366] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116368] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116370] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116372] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116374] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116376] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116378] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116379] Stack: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116380] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116382] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116384] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116387] Call Trace: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116422] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116468] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116518] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116565] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116612] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116627] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116638] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116650] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116662] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116712] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116723] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116738] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116769] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116786] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116815] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116829] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116844] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116895] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116899] [] ? strlcpy+0x4a/0x60 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116915] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116930] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116944] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.116959] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117003] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117051] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117105] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117159] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117212] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117264] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117268] [] ? thread_return+0x4e/0x7d0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117320] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117323] [] ? kthread+0x9e/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117326] [] ? child_rip+0xa/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117328] [] ? kthread+0x0/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117331] [] ? child_rip+0x0/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117332] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117347] Call Trace: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117390] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117436] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117486] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117533] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117579] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117595] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117605] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117617] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117629] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117679] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117690] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117705] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117734] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117750] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117779] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117793] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117808] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117859] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117863] [] ? strlcpy+0x4a/0x60 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117879] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117893] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117908] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117923] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.117967] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118007] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118061] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118115] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118168] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118220] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118224] [] ? thread_return+0x4e/0x7d0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118275] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118279] [] ? kthread+0x9e/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118281] [] ? child_rip+0xa/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118284] [] ? kthread+0x0/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.118286] [] ? child_rip+0x0/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119266] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119268] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119307] CPU 7 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119308] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119345] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119347] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119350] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119354] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119355] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119357] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119358] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119360] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119362] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119364] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119366] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119367] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119369] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119371] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119373] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119374] Stack: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119375] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119377] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119379] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119382] Call Trace: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119415] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119462] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119512] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119558] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119605] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119621] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119631] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119643] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119654] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119669] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119685] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119714] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119729] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119744] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119794] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119798] [] ? strlcpy+0x4a/0x60 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119814] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119829] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119843] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119858] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119902] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119942] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.119996] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120051] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120104] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120156] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120160] [] ? thread_return+0x4e/0x7d0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120211] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120215] [] ? kthread+0x9e/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120218] [] ? child_rip+0xa/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120220] [] ? kthread+0x0/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120223] [] ? child_rip+0x0/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120224] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120239] Call Trace: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120282] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120328] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120378] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120424] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120471] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120486] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120497] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120509] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120519] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120534] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120550] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120579] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120593] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120609] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120659] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120663] [] ? strlcpy+0x4a/0x60 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120679] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120693] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120708] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120723] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120767] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120807] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120861] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120915] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.120969] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.121021] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.121024] [] ? thread_return+0x4e/0x7d0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.121076] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.121079] [] ? kthread+0x9e/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.121082] [] ? child_rip+0xa/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.121084] [] ? kthread+0x0/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483678.121087] [] ? child_rip+0x0/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.641560] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.738828] CPU 4 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.740979] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.838751] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.840616] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.854161] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.862888] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.868781] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.876712] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.884625] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.892521] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.900504] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.908486] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.917464] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.923852] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.931783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.939722] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.950295] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.959855] Stack: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.962282] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.970220] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.978800] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.987606] Call Trace: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.990518] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483680.997981] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.006028] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.013687] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.021219] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.028780] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.036384] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.043770] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.050909] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.057840] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.065736] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.072934] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.081249] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.087568] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.094569] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.102516] [] ? strlcpy+0x4a/0x60 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.108143] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.114998] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.122409] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.129593] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.136911] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.144509] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.152564] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.160001] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.166883] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.174829] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.181844] [] ? thread_return+0x4e/0x7d0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.188209] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.195105] [] ? kthread+0x9e/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.200720] [] ? child_rip+0xa/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.206473] [] ? kthread+0x0/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.211978] [] ? child_rip+0x0/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.217723] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.239165] Call Trace: Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.242050] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.249409] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.257509] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.265194] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.272691] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.280304] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.287926] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.295363] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.302478] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.309363] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.317174] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.324328] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.332600] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.338927] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.345948] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.353867] [] ? strlcpy+0x4a/0x60 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.359478] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.366263] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.373586] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.380805] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.388219] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.395773] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.403856] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.411258] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.418242] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.426093] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.433185] [] ? thread_return+0x4e/0x7d0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.439536] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.446455] [] ? kthread+0x9e/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.452115] [] ? child_rip+0xa/0x20 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.457877] [] ? kthread+0x0/0xc0 Aug 25 20:23:39 feral28.ccs.ornl.gov kernel: [1483681.463381] [] ? child_rip+0x0/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.115876] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118876] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118877] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118919] CPU 5 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118920] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118956] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118958] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118961] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118965] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118966] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118968] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118970] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118971] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118973] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118975] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118976] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118978] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118980] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118983] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118984] Stack: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118985] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118987] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118990] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.118992] Call Trace: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119036] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119083] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119132] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119179] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119226] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119242] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119252] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119264] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119275] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119290] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119306] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119335] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119350] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119365] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119415] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119419] [] ? strlcpy+0x4a/0x60 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119435] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119450] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119465] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119480] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119524] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119564] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119618] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119672] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119726] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119778] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119781] [] ? thread_return+0x4e/0x7d0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119833] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119837] [] ? kthread+0x9e/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119839] [] ? child_rip+0xa/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119842] [] ? kthread+0x0/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119844] [] ? child_rip+0x0/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119845] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119861] Call Trace: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119903] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119949] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.119999] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120046] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120092] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120108] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120118] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120130] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120141] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120156] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120172] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120201] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120215] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120230] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120281] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120284] [] ? strlcpy+0x4a/0x60 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120301] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120316] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120330] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120345] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120389] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120429] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120483] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120537] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120591] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120643] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120646] [] ? thread_return+0x4e/0x7d0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120698] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120702] [] ? kthread+0x9e/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120704] [] ? child_rip+0xa/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120707] [] ? kthread+0x0/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.120709] [] ? child_rip+0x0/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122876] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122877] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122919] CPU 6 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122920] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122956] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122958] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122961] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122966] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122967] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122969] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122971] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122972] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122974] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122976] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122978] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122980] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122984] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122986] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122987] Stack: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122988] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122990] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122992] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.122995] Call Trace: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123030] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123076] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123126] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123173] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123219] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123235] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123246] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123257] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123269] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123320] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123330] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123345] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123377] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123393] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123422] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123437] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123452] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123502] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123506] [] ? strlcpy+0x4a/0x60 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123522] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123537] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123552] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123567] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123611] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123651] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123705] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123759] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123813] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123865] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123868] [] ? thread_return+0x4e/0x7d0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123920] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123923] [] ? kthread+0x9e/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123926] [] ? child_rip+0xa/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123929] [] ? kthread+0x0/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123931] [] ? child_rip+0x0/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123932] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123948] Call Trace: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.123990] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124037] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124086] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124133] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124180] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124195] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124206] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124218] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124230] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124280] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124290] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124306] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124336] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124353] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124381] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124396] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124411] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124461] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124465] [] ? strlcpy+0x4a/0x60 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124481] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124496] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124511] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124526] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124569] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124610] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124664] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124718] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124771] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124823] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124826] [] ? thread_return+0x4e/0x7d0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124878] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124882] [] ? kthread+0x9e/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124885] [] ? child_rip+0xa/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124887] [] ? kthread+0x0/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.124890] [] ? child_rip+0x0/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125875] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125876] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125916] CPU 7 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125917] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125953] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125954] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125957] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125962] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125963] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125965] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125967] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125968] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125970] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125972] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125974] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125975] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125977] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125978] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125980] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125982] Stack: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125982] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125984] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125987] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.125989] Call Trace: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126023] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126069] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126119] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126166] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126212] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126228] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126238] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126250] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126261] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126276] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126292] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126321] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126336] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126351] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126401] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126405] [] ? strlcpy+0x4a/0x60 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126421] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126436] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126450] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126465] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126509] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126549] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126603] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126657] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126711] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126763] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126766] [] ? thread_return+0x4e/0x7d0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126818] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126821] [] ? kthread+0x9e/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126824] [] ? child_rip+0xa/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126827] [] ? kthread+0x0/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126829] [] ? child_rip+0x0/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126831] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126846] Call Trace: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126889] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126935] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.126985] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127031] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127078] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127093] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127104] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127116] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127126] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127141] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127157] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127186] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127201] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127216] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127266] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127270] [] ? strlcpy+0x4a/0x60 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127286] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127301] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127316] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127330] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127374] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127415] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127469] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127523] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127576] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127628] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127631] [] ? thread_return+0x4e/0x7d0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127683] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127686] [] ? kthread+0x9e/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127689] [] ? child_rip+0xa/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127692] [] ? kthread+0x0/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483762.127694] [] ? child_rip+0x0/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.649884] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.749962] CPU 4 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.752077] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.849788] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.851678] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.865092] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.873735] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.879620] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.887559] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.895403] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.903290] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.911185] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.919064] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.927887] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.934230] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.942100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.949953] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.957856] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.967313] Stack: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.969739] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.977609] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.986102] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.994969] Call Trace: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483764.997888] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.005206] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.013235] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.020832] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.028259] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.035814] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.043418] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.050708] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.057744] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.064595] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.072370] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.079506] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.087727] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.094116] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.101102] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.108940] [] ? strlcpy+0x4a/0x60 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.114554] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.121390] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.128749] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.135966] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.143243] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.150874] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.158923] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.166358] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.173274] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.181160] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.188253] [] ? thread_return+0x4e/0x7d0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.194532] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.201417] [] ? kthread+0x9e/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.207000] [] ? child_rip+0xa/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.212659] [] ? kthread+0x0/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.218127] [] ? child_rip+0x0/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.223820] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.245427] Call Trace: Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.248295] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.255647] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.263667] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.271274] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.278736] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.286228] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.293850] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.301209] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.308246] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.315175] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.322985] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.330122] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.338420] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.344756] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.351734] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.359642] [] ? strlcpy+0x4a/0x60 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.365332] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.372125] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.379545] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.386677] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.393944] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.401549] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.409624] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.417077] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.424061] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.431947] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.439075] [] ? thread_return+0x4e/0x7d0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.445382] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.452249] [] ? kthread+0x9e/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.457889] [] ? child_rip+0xa/0x20 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.463585] [] ? kthread+0x0/0xc0 Aug 25 20:25:03 feral28.ccs.ornl.gov kernel: [1483765.469069] [] ? child_rip+0x0/0x20 Aug 25 20:25:49 feral28.ccs.ornl.gov kernel: [1483811.855722] Lustre: sultan-MDT000b: haven't heard from client 1561233c-aa4b-81d5-e0e4-de3b3609634b (at 2@gni1) in 305 seconds. I think it's dead, and I am evicting it. exp ffff880b9dbd5000, cur 1472171149 expire 1472170927 last 1472170844 Aug 25 20:25:49 feral28.ccs.ornl.gov kernel: [1483811.878440] Lustre: Skipped 1 previous similar message Aug 25 20:26:02 feral28.ccs.ornl.gov kernel: [1483825.135948] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 20:26:02 feral28.ccs.ornl.gov kernel: [1483825.146971] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 37 previous similar messages Aug 25 20:26:02 feral28.ccs.ornl.gov kernel: [1483825.158034] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.119@o2ib1 (217): c: 63, oc: 0, rc: 63 Aug 25 20:26:02 feral28.ccs.ornl.gov kernel: [1483825.171445] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 37 previous similar messages Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.122484] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125484] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125486] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125528] CPU 5 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125529] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125565] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125567] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125570] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125574] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125575] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125577] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125579] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125580] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125582] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125584] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125585] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125587] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125590] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125592] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125594] Stack: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125594] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125596] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125599] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125601] Call Trace: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125645] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125691] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125741] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125788] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125834] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125850] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125861] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125873] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125883] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125899] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125915] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125944] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125959] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.125974] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126024] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126028] [] ? strlcpy+0x4a/0x60 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126044] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126059] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126074] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126089] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126133] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126173] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126227] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126281] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126335] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126387] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126390] [] ? thread_return+0x4e/0x7d0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126442] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126446] [] ? kthread+0x9e/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126449] [] ? child_rip+0xa/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126451] [] ? kthread+0x0/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126454] [] ? child_rip+0x0/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126455] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126470] Call Trace: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126513] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126559] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126609] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126655] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126702] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126718] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126728] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126740] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126750] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126766] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126782] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126811] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126825] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126840] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126891] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126895] [] ? strlcpy+0x4a/0x60 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126911] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126926] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126940] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126955] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.126999] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127039] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127093] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127147] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127201] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127253] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127256] [] ? thread_return+0x4e/0x7d0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127308] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127312] [] ? kthread+0x9e/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127315] [] ? child_rip+0xa/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127317] [] ? kthread+0x0/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.127320] [] ? child_rip+0x0/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129484] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129486] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129529] CPU 6 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129530] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129565] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129567] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129570] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129574] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129576] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129578] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129579] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129581] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129582] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129584] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129586] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129588] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129592] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129594] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129595] Stack: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129596] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129598] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129600] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129603] Call Trace: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129638] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129684] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129734] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129781] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129827] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129843] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129854] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129865] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129878] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129928] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129938] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129954] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.129985] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130002] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130030] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130045] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130060] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130111] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130115] [] ? strlcpy+0x4a/0x60 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130131] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130146] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130161] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130175] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130219] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130260] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130314] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130368] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130421] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130473] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130477] [] ? thread_return+0x4e/0x7d0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130529] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130532] [] ? kthread+0x9e/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130535] [] ? child_rip+0xa/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130538] [] ? kthread+0x0/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130540] [] ? child_rip+0x0/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130541] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130556] Call Trace: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130599] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130646] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130695] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130742] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130788] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130804] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130815] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130826] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130838] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130888] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130899] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130914] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130943] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130960] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.130988] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131003] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131018] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131068] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131072] [] ? strlcpy+0x4a/0x60 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131089] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131103] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131118] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131133] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131177] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131217] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131271] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131325] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131378] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131430] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131434] [] ? thread_return+0x4e/0x7d0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131486] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131489] [] ? kthread+0x9e/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131492] [] ? child_rip+0xa/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131494] [] ? kthread+0x0/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.131497] [] ? child_rip+0x0/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132484] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132485] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132527] CPU 7 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132528] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132565] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132566] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132569] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132574] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132575] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132577] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132579] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132580] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132582] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132584] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132586] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132588] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132591] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132593] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132595] Stack: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132595] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132597] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132600] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132602] Call Trace: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132645] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132692] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132741] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132788] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132835] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132850] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132861] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132873] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132883] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132898] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132915] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132943] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132958] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.132973] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133023] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133027] [] ? strlcpy+0x4a/0x60 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133043] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133058] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133073] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133088] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133132] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133172] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133226] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133280] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133334] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133386] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133389] [] ? thread_return+0x4e/0x7d0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133441] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133444] [] ? kthread+0x9e/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133447] [] ? child_rip+0xa/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133450] [] ? kthread+0x0/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133452] [] ? child_rip+0x0/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133453] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133468] Call Trace: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133511] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133557] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133607] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133653] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133700] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133716] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133726] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133738] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133749] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133764] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133780] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133808] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133823] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133838] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133888] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133892] [] ? strlcpy+0x4a/0x60 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133908] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133923] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133937] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133952] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.133996] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134036] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134090] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134144] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134198] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134249] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134253] [] ? thread_return+0x4e/0x7d0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134304] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134308] [] ? kthread+0x9e/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134311] [] ? child_rip+0xa/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134313] [] ? kthread+0x0/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483846.134316] [] ? child_rip+0x0/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.657375] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.754524] CPU 4 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.756657] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.854256] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.856136] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.869666] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.878445] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.884304] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.892287] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.900217] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.908208] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.916164] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.924095] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.933031] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.939417] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.947433] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.955312] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.963261] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.972758] Stack: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.975152] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.983047] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483848.991628] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.000459] Call Trace: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.003396] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.010783] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.018830] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.026557] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.034132] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.041710] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.049325] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.056710] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.063859] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.070744] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.078642] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.085874] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.094198] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.100576] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.107615] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.115611] [] ? strlcpy+0x4a/0x60 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.121231] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.128024] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.135400] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.142602] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.149903] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.157440] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.165540] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.172959] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.179944] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.187795] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.194905] [] ? thread_return+0x4e/0x7d0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.201228] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.208139] [] ? kthread+0x9e/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.213737] [] ? child_rip+0xa/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.219475] [] ? kthread+0x0/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.224995] [] ? child_rip+0x0/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.230680] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.252139] Call Trace: Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.255093] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.262533] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.270605] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.278229] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.285724] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.293234] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.300883] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.308217] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.315314] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.322174] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.330026] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.337190] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.345487] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.351780] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.358862] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.366815] [] ? strlcpy+0x4a/0x60 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.372486] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.379349] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.386681] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.393910] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.401227] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.408850] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.416890] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.424351] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.431330] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.439241] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.446325] [] ? thread_return+0x4e/0x7d0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.452684] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.459551] [] ? kthread+0x9e/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.467862] [] ? child_rip+0xa/0x20 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.473600] [] ? kthread+0x0/0xc0 Aug 25 20:26:27 feral28.ccs.ornl.gov kernel: [1483849.479137] [] ? child_rip+0x0/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.129093] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132093] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132095] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132137] CPU 5 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132137] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132173] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132175] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132178] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132182] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132184] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132185] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132187] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132189] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132190] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132192] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132194] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132195] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132197] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132199] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132201] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132202] Stack: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132203] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132205] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132207] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132210] Call Trace: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132253] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132300] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132350] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132396] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132443] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132459] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132470] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132482] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132492] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132507] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132523] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132553] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132568] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132583] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132633] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132637] [] ? strlcpy+0x4a/0x60 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132653] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132668] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132683] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132698] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132742] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132782] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132836] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132891] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132944] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.132996] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133000] [] ? thread_return+0x4e/0x7d0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133052] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133055] [] ? kthread+0x9e/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133058] [] ? child_rip+0xa/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133060] [] ? kthread+0x0/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133063] [] ? child_rip+0x0/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133064] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133079] Call Trace: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133122] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133168] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133218] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133264] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133311] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133326] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133337] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133349] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133359] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133374] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133390] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133419] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133434] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133448] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133498] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133503] [] ? strlcpy+0x4a/0x60 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133519] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133533] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133548] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133563] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133607] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133647] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133701] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133755] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133809] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133861] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133864] [] ? thread_return+0x4e/0x7d0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133916] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133920] [] ? kthread+0x9e/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133922] [] ? child_rip+0xa/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133925] [] ? kthread+0x0/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.133927] [] ? child_rip+0x0/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136093] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136095] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136137] CPU 6 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136138] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136174] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136175] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136178] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136183] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136184] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136186] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136187] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136189] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136191] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136193] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136195] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136196] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136200] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136202] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136203] Stack: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136204] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136206] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136208] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136211] Call Trace: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136246] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136293] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136343] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136389] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136436] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136452] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136463] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136474] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136486] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136537] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136547] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136563] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136592] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136609] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136638] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136652] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136668] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136718] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136722] [] ? strlcpy+0x4a/0x60 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136738] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136753] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136768] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136783] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136827] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136867] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136921] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.136975] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137029] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137081] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137084] [] ? thread_return+0x4e/0x7d0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137136] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137140] [] ? kthread+0x9e/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137142] [] ? child_rip+0xa/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137145] [] ? kthread+0x0/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137147] [] ? child_rip+0x0/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137149] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137164] Call Trace: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137207] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137253] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137303] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137349] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137396] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137412] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137422] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137434] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137446] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137496] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137507] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137522] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137551] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137568] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137596] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137611] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137626] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137676] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137680] [] ? strlcpy+0x4a/0x60 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137696] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137711] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137725] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137740] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137784] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137825] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137879] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137933] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.137987] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.138039] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.138042] [] ? thread_return+0x4e/0x7d0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.138094] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.138097] [] ? kthread+0x9e/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.138100] [] ? child_rip+0xa/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.138103] [] ? kthread+0x0/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.138105] [] ? child_rip+0x0/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139092] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139094] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139133] CPU 7 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139134] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139170] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139172] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139175] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139179] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139181] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139182] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139184] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139186] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139187] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139189] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139191] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139193] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139196] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139198] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139199] Stack: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139200] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139202] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139204] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139206] Call Trace: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139240] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139287] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139336] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139383] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139429] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139445] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139456] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139468] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139478] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139493] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139509] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139538] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139553] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139568] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139618] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139622] [] ? strlcpy+0x4a/0x60 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139638] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139653] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139668] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139683] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139727] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139767] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139821] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139875] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139929] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139981] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.139984] [] ? thread_return+0x4e/0x7d0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140036] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140039] [] ? kthread+0x9e/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140042] [] ? child_rip+0xa/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140045] [] ? kthread+0x0/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140047] [] ? child_rip+0x0/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140048] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140063] Call Trace: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140106] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140152] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140202] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140248] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140295] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140310] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140321] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140333] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140343] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140358] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140374] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140403] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140417] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140432] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140482] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140486] [] ? strlcpy+0x4a/0x60 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140503] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140517] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140532] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140547] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140591] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140631] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140685] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140739] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140792] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140844] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140848] [] ? thread_return+0x4e/0x7d0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140899] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140903] [] ? kthread+0x9e/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140906] [] ? child_rip+0xa/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140909] [] ? kthread+0x0/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483930.140911] [] ? child_rip+0x0/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.662521] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.760040] CPU 4 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.762156] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.859580] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.861472] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.874983] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.883615] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.889526] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.897472] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.905370] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.913352] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.921308] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.929273] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.938174] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.944527] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.952483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.960475] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.968362] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.977868] Stack: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.980261] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.988226] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483932.996727] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.005568] Call Trace: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.008496] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.015936] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.024084] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.031709] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.039189] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.046785] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.054467] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.061844] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.068976] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.075940] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.083811] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.091000] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.099289] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.105686] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.112732] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.120642] [] ? strlcpy+0x4a/0x60 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.126297] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.133091] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.140473] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.147703] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.154977] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.162505] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.170588] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.177947] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.184879] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.192808] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.199909] [] ? thread_return+0x4e/0x7d0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.206303] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.213162] [] ? kthread+0x9e/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.218786] [] ? child_rip+0xa/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.224515] [] ? kthread+0x0/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.232722] [] ? child_rip+0x0/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.238467] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.259987] Call Trace: Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.262898] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.270311] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.278366] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.286033] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.293513] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.301055] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.308642] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.315970] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.323032] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.330012] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.337883] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.345133] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.353473] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.359843] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.366848] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.374793] [] ? strlcpy+0x4a/0x60 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.380439] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.387231] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.394590] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.401810] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.409162] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.416768] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.424851] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.432305] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.439203] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.447062] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.454147] [] ? thread_return+0x4e/0x7d0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.460444] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.467397] [] ? kthread+0x9e/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.472997] [] ? child_rip+0xa/0x20 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.478699] [] ? kthread+0x0/0xc0 Aug 25 20:27:51 feral28.ccs.ornl.gov kernel: [1483933.484229] [] ? child_rip+0x0/0x20 Aug 25 20:28:10 feral28.ccs.ornl.gov kernel: [1483952.596159] Lustre: sultan-MDT000b: haven't heard from client 165f3703-4728-d396-aa69-684cbccc927f (at 13@gni1) in 338 seconds. I think it's dead, and I am evicting it. exp ffff880ba2efa000, cur 1472171290 expire 1472171068 last 1472170952 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.135702] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138702] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138704] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138746] CPU 5 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138747] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138783] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138785] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138788] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138792] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138794] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138795] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138797] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138798] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138800] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138802] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138804] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138806] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138808] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138809] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138811] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138813] Stack: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138814] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138816] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138818] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138820] Call Trace: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138864] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138911] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.138960] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139007] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139054] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139069] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139080] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139092] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139103] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139118] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139134] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139163] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139178] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139193] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139244] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139248] [] ? strlcpy+0x4a/0x60 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139264] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139279] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139293] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139308] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139352] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139392] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139447] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139501] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139554] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139606] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139610] [] ? thread_return+0x4e/0x7d0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139662] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139665] [] ? kthread+0x9e/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139668] [] ? child_rip+0xa/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139671] [] ? kthread+0x0/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139673] [] ? child_rip+0x0/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139674] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139690] Call Trace: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139732] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139779] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139828] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139875] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139921] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139937] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139948] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139960] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139970] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.139985] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140001] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140030] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140045] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140060] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140110] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140114] [] ? strlcpy+0x4a/0x60 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140130] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140145] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140160] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140174] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140218] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140259] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140313] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140367] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140421] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140473] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140476] [] ? thread_return+0x4e/0x7d0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140528] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140531] [] ? kthread+0x9e/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140534] [] ? child_rip+0xa/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140537] [] ? kthread+0x0/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.140539] [] ? child_rip+0x0/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142702] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142704] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142746] CPU 6 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142747] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142783] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142785] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142788] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142792] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142794] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142795] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142797] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142798] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142800] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142802] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142804] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142806] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142807] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142809] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142811] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142812] Stack: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142813] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142815] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142818] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142820] Call Trace: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142855] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142901] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142951] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.142998] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143045] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143060] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143071] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143083] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143095] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143145] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143156] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143171] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143202] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143219] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143248] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143262] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143277] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143328] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143332] [] ? strlcpy+0x4a/0x60 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143348] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143363] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143378] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143393] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143436] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143477] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143531] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143585] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143638] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143690] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143694] [] ? thread_return+0x4e/0x7d0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143745] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143749] [] ? kthread+0x9e/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143752] [] ? child_rip+0xa/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143754] [] ? kthread+0x0/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143756] [] ? child_rip+0x0/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143758] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143773] Call Trace: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143816] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143862] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143912] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.143958] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144005] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144020] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144031] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144043] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144055] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144105] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144116] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144131] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144160] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144176] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144204] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144219] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144234] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144284] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144288] [] ? strlcpy+0x4a/0x60 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144304] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144319] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144334] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144349] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144393] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144433] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144487] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144541] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144594] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144646] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144650] [] ? thread_return+0x4e/0x7d0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144701] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144705] [] ? kthread+0x9e/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144708] [] ? child_rip+0xa/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144710] [] ? kthread+0x0/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.144712] [] ? child_rip+0x0/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145701] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145703] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145742] CPU 7 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145743] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145779] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145781] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145784] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145788] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145790] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145791] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145793] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145795] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145796] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145798] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145800] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145802] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145805] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145807] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145808] Stack: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145809] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145811] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145813] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145816] Call Trace: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145850] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145896] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145946] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.145992] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146039] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146055] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146065] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146077] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146088] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146103] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146119] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146148] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146162] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146177] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146227] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146231] [] ? strlcpy+0x4a/0x60 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146248] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146262] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146277] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146292] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146336] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146376] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146430] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146484] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146538] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146589] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146593] [] ? thread_return+0x4e/0x7d0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146645] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146648] [] ? kthread+0x9e/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146651] [] ? child_rip+0xa/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146654] [] ? kthread+0x0/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146656] [] ? child_rip+0x0/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146657] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146672] Call Trace: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146715] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146761] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146811] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146858] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146904] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146920] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146930] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146942] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146953] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146968] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.146984] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147013] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147027] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147042] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147092] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147096] [] ? strlcpy+0x4a/0x60 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147112] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147127] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147142] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147157] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147200] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147241] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147295] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147349] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147402] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147454] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147457] [] ? thread_return+0x4e/0x7d0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147509] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147513] [] ? kthread+0x9e/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147515] [] ? child_rip+0xa/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147518] [] ? kthread+0x0/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484014.147520] [] ? child_rip+0x0/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.648632] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.745797] CPU 4 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.747903] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.845199] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.847062] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.860530] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.869171] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.875057] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.882890] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.890794] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.898681] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.906551] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.914509] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.923315] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.929639] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.937528] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.945432] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.953259] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.962636] Stack: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.967706] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.975576] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.984060] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.992892] Call Trace: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484016.995786] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.003174] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.011202] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.018800] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.026219] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.033729] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.041298] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.048623] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.055625] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.062502] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.070336] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.077467] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.085704] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.092037] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.099025] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.106891] [] ? strlcpy+0x4a/0x60 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.112538] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.119355] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.126776] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.134031] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.141348] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.148997] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.157019] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.164413] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.171354] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.179265] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.186349] [] ? thread_return+0x4e/0x7d0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.192631] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.199499] [] ? kthread+0x9e/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.205097] [] ? child_rip+0xa/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.210773] [] ? kthread+0x0/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.216268] [] ? child_rip+0x0/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.221926] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.243550] Call Trace: Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.246443] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.253864] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.261920] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.269500] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.276903] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.284386] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.291999] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.299272] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.306326] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.313229] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.321054] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.328219] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.336438] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.342766] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.349742] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.357619] [] ? strlcpy+0x4a/0x60 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.363239] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.370076] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.377451] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.384608] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.391954] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.399569] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.407676] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.415148] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.422107] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.430045] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.437188] [] ? thread_return+0x4e/0x7d0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.443546] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.450424] [] ? kthread+0x9e/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.456014] [] ? child_rip+0xa/0x20 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.461725] [] ? kthread+0x0/0xc0 Aug 25 20:29:15 feral28.ccs.ornl.gov kernel: [1484017.467255] [] ? child_rip+0x0/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.142311] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145311] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145313] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145355] CPU 5 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145356] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145392] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145394] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145397] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145401] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145403] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145404] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145406] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145407] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145409] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145411] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145413] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145414] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145416] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145418] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145419] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145421] Stack: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145422] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145424] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145426] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145429] Call Trace: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145472] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145519] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145569] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145615] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145662] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145678] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145688] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145700] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145711] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145727] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145743] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145772] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145786] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145801] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145852] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145856] [] ? strlcpy+0x4a/0x60 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145873] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145888] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145903] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145917] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.145961] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146002] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146056] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146110] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146163] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146215] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146219] [] ? thread_return+0x4e/0x7d0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146271] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146274] [] ? kthread+0x9e/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146277] [] ? child_rip+0xa/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146279] [] ? kthread+0x0/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146282] [] ? child_rip+0x0/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146283] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146298] Call Trace: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146341] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146387] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146437] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146484] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146530] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146546] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146556] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146568] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146578] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146594] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146610] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146638] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146653] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146668] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146718] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146722] [] ? strlcpy+0x4a/0x60 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146738] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146753] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146768] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146783] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146827] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146867] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146921] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.146975] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.147029] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.147081] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.147084] [] ? thread_return+0x4e/0x7d0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.147136] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.147140] [] ? kthread+0x9e/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.147142] [] ? child_rip+0xa/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.147145] [] ? kthread+0x0/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.147147] [] ? child_rip+0x0/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149311] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149313] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149355] CPU 6 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149356] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149392] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149394] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149396] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149401] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149403] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149404] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149406] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149407] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149409] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149411] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149413] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149414] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149416] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149418] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149420] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149421] Stack: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149422] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149424] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149426] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149429] Call Trace: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149463] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149510] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149560] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149607] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149653] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149669] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149679] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149691] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149703] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149754] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149764] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149780] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149809] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149826] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149855] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149869] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149884] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149935] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149939] [] ? strlcpy+0x4a/0x60 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149955] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149970] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149984] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.149999] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150043] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150084] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150138] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150192] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150246] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150297] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150301] [] ? thread_return+0x4e/0x7d0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150353] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150356] [] ? kthread+0x9e/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150359] [] ? child_rip+0xa/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150362] [] ? kthread+0x0/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150364] [] ? child_rip+0x0/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150365] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150380] Call Trace: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150423] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150470] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150519] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150566] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150612] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150628] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150638] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150650] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150662] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150713] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150723] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150739] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150768] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150784] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150813] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150827] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150842] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150893] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150897] [] ? strlcpy+0x4a/0x60 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150913] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150928] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150942] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.150957] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151001] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151041] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151095] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151149] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151203] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151255] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151258] [] ? thread_return+0x4e/0x7d0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151310] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151314] [] ? kthread+0x9e/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151316] [] ? child_rip+0xa/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151319] [] ? kthread+0x0/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.151321] [] ? child_rip+0x0/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152310] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152312] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152351] CPU 7 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152352] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152388] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152389] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152392] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152396] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152398] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152400] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152401] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152403] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152405] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152407] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152409] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152410] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152412] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152414] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152416] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152417] Stack: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152418] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152420] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152422] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152424] Call Trace: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152458] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152505] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152554] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152601] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152647] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152663] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152674] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152686] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152696] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152711] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152727] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152756] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152771] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152786] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152836] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152840] [] ? strlcpy+0x4a/0x60 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152856] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152871] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152886] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152901] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152945] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.152985] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153039] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153093] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153147] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153199] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153202] [] ? thread_return+0x4e/0x7d0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153254] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153258] [] ? kthread+0x9e/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153260] [] ? child_rip+0xa/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153263] [] ? kthread+0x0/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153265] [] ? child_rip+0x0/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153267] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153282] Call Trace: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153324] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153371] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153420] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153467] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153514] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153529] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153540] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153551] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153562] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153577] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153593] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153622] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153636] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153651] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153701] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153705] [] ? strlcpy+0x4a/0x60 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153721] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153736] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153751] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153766] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153810] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153850] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153904] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.153958] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.154012] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.154063] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.154067] [] ? thread_return+0x4e/0x7d0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.154118] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.154122] [] ? kthread+0x9e/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.154125] [] ? child_rip+0xa/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.154127] [] ? kthread+0x0/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484098.154129] [] ? child_rip+0x0/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.674618] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.771967] CPU 4 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.774090] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.874366] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.876222] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.889733] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.898445] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.904346] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.912234] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.920222] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.928181] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.936110] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.944092] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.953029] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.959443] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.967329] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.975234] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.983139] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.992628] Stack: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484100.995046] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.002968] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.011512] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.020275] Call Trace: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.023178] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.030591] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.038663] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.046357] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.053897] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.061371] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.069010] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.076405] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.083511] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.090492] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.098405] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.105604] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.113875] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.120229] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.127266] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.135237] [] ? strlcpy+0x4a/0x60 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.140849] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.147633] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.155069] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.162307] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.169623] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.177238] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.185339] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.192774] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.199734] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.207611] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.214722] [] ? thread_return+0x4e/0x7d0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.221053] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.227948] [] ? kthread+0x9e/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.233587] [] ? child_rip+0xa/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.239317] [] ? kthread+0x0/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.244847] [] ? child_rip+0x0/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.250591] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.272121] Call Trace: Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.275040] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.282444] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.290491] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.298149] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.305629] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.313173] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.320836] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.328179] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.335365] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.342337] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.350121] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.357414] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.365737] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.372116] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.379128] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.387108] [] ? strlcpy+0x4a/0x60 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.392701] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.399521] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.406897] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.414099] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.421425] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.429049] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.437193] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.444619] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.451579] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.459516] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.466575] [] ? thread_return+0x4e/0x7d0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.472907] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.479774] [] ? kthread+0x9e/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.485382] [] ? child_rip+0xa/0x20 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.491163] [] ? kthread+0x0/0xc0 Aug 25 20:30:39 feral28.ccs.ornl.gov kernel: [1484101.496698] [] ? child_rip+0x0/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.148920] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.151920] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.151921] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.151964] CPU 5 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.151965] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152001] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152002] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152005] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152010] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152011] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152013] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152014] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152016] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152018] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152020] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152022] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152023] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152027] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152029] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152030] Stack: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152031] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152033] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152035] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152038] Call Trace: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152082] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152128] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152178] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152225] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152271] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152287] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152298] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152310] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152320] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152336] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152352] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152381] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152396] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152411] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152461] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152465] [] ? strlcpy+0x4a/0x60 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152482] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152496] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152511] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152526] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152570] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152610] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152665] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152719] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152773] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152825] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152828] [] ? thread_return+0x4e/0x7d0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152880] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152883] [] ? kthread+0x9e/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152886] [] ? child_rip+0xa/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152889] [] ? kthread+0x0/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152891] [] ? child_rip+0x0/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152892] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152908] Call Trace: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152950] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.152997] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153046] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153093] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153139] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153155] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153166] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153177] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153188] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153203] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153219] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153248] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153263] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153278] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153328] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153332] [] ? strlcpy+0x4a/0x60 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153348] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153363] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153378] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153392] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153436] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153476] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153531] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153585] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153638] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153690] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153693] [] ? thread_return+0x4e/0x7d0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153745] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153749] [] ? kthread+0x9e/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153752] [] ? child_rip+0xa/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153755] [] ? kthread+0x0/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.153757] [] ? child_rip+0x0/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.155920] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.155922] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.155964] CPU 6 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.155965] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156001] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156003] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156006] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156010] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156012] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156013] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156015] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156017] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156018] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156020] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156022] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156024] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156027] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156029] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156031] Stack: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156032] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156034] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156036] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156039] Call Trace: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156073] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156120] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156170] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156216] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156263] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156279] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156289] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156301] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156313] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156364] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156374] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156390] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156421] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156437] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156466] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156481] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156496] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156546] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156550] [] ? strlcpy+0x4a/0x60 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156566] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156581] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156596] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156611] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156654] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156695] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156749] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156803] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156856] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156908] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156912] [] ? thread_return+0x4e/0x7d0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156963] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156967] [] ? kthread+0x9e/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156970] [] ? child_rip+0xa/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156972] [] ? kthread+0x0/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156975] [] ? child_rip+0x0/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156976] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.156991] Call Trace: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157034] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157080] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157130] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157176] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157223] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157238] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157249] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157261] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157273] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157323] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157334] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157349] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157380] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157397] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157425] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157440] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157455] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157505] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157509] [] ? strlcpy+0x4a/0x60 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157525] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157540] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157555] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157570] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157613] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157654] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157708] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157762] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157816] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157867] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157871] [] ? thread_return+0x4e/0x7d0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157922] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157926] [] ? kthread+0x9e/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157929] [] ? child_rip+0xa/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157932] [] ? kthread+0x0/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.157934] [] ? child_rip+0x0/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.158919] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.158921] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.158960] CPU 7 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.158961] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.158996] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.158998] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159001] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159005] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159007] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159008] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159010] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159011] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159013] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159015] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159017] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159019] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159024] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159026] Stack: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159027] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159029] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159031] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159033] Call Trace: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159067] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159114] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159163] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159210] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159257] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159272] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159283] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159295] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159305] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159321] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159337] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159366] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159380] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159395] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159446] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159450] [] ? strlcpy+0x4a/0x60 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159466] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159481] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159496] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159510] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159554] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159594] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159648] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159702] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159756] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159808] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159811] [] ? thread_return+0x4e/0x7d0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159863] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159866] [] ? kthread+0x9e/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159869] [] ? child_rip+0xa/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159872] [] ? kthread+0x0/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159874] [] ? child_rip+0x0/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159875] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159890] Call Trace: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159933] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.159979] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160029] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160075] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160122] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160137] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160148] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160160] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160170] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160185] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160201] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160230] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160245] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160260] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160310] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160313] [] ? strlcpy+0x4a/0x60 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160330] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160344] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160359] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160374] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160418] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160458] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160512] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160566] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160620] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160671] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160675] [] ? thread_return+0x4e/0x7d0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160726] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160730] [] ? kthread+0x9e/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160733] [] ? child_rip+0xa/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160736] [] ? kthread+0x0/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484182.160738] [] ? child_rip+0x0/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.684225] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.781652] CPU 4 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.783758] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.881400] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.883280] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.896810] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.905537] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.911490] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.919397] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.927378] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.935292] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.943169] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.951049] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.960070] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.966458] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.974440] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.982353] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.990241] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484184.999748] Stack: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.002166] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.010122] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.018657] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.027515] Call Trace: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.030436] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.037902] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.046017] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.053702] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.061259] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.068812] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.076484] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.083863] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.090994] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.097924] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.105768] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.112940] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.121263] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.127600] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.134638] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.142582] [] ? strlcpy+0x4a/0x60 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.148253] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.155116] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.162553] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.169816] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.177117] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.184729] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.192797] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.200189] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.207157] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.215042] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.222127] [] ? thread_return+0x4e/0x7d0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.228468] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.235457] [] ? kthread+0x9e/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.241047] [] ? child_rip+0xa/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.246715] [] ? kthread+0x0/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.252217] [] ? child_rip+0x0/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.257980] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.279370] Call Trace: Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.282308] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.289686] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.297784] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.305529] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.313026] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.320605] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.328210] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.335543] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.342655] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.349543] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.357422] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.364612] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.372900] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.379288] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.386404] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.394349] [] ? strlcpy+0x4a/0x60 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.399970] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.406797] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.414181] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.421426] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.428797] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.436377] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.444442] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.451843] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.458759] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.466575] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.473669] [] ? thread_return+0x4e/0x7d0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.479999] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.486954] [] ? kthread+0x9e/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.492543] [] ? child_rip+0xa/0x20 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.498213] [] ? kthread+0x0/0xc0 Aug 25 20:32:03 feral28.ccs.ornl.gov kernel: [1484185.503803] [] ? child_rip+0x0/0x20 Aug 25 20:32:09 feral28.ccs.ornl.gov kernel: [1484192.210767] Lustre: sultan-MDT000b: Client f680e5fb-3752-3b31-4864-2e3f707d1adc (at 12@gni1) reconnecting Aug 25 20:32:09 feral28.ccs.ornl.gov kernel: [1484192.221127] Lustre: Skipped 10 previous similar messages Aug 25 20:32:09 feral28.ccs.ornl.gov kernel: [1484192.227001] Lustre: sultan-MDT000b: Connection restored to 560c2e5a-a98f-72ed-fa5f-899936c4b413 (at 12@gni1) Aug 25 20:32:09 feral28.ccs.ornl.gov kernel: [1484192.237639] Lustre: Skipped 11 previous similar messages Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.155529] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158529] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158531] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158573] CPU 5 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158574] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158610] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158612] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158615] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158619] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158621] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158622] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158624] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158625] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158627] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158629] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158631] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158632] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158636] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158638] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158639] Stack: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158640] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158642] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158644] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158647] Call Trace: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158690] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158737] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158786] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158833] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158880] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158895] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158906] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158918] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158929] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158944] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158960] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.158989] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159004] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159019] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159070] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159074] [] ? strlcpy+0x4a/0x60 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159090] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159104] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159119] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159134] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159178] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159219] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159273] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159327] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159380] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159432] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159436] [] ? thread_return+0x4e/0x7d0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159487] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159491] [] ? kthread+0x9e/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159493] [] ? child_rip+0xa/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159496] [] ? kthread+0x0/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159498] [] ? child_rip+0x0/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159500] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159515] Call Trace: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159557] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159604] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159653] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159700] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159747] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159762] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159773] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159785] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159795] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159810] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159826] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159855] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159870] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159885] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159935] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159939] [] ? strlcpy+0x4a/0x60 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159955] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159970] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159984] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.159999] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160043] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160083] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160138] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160192] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160245] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160297] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160301] [] ? thread_return+0x4e/0x7d0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160352] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160356] [] ? kthread+0x9e/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160359] [] ? child_rip+0xa/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160362] [] ? kthread+0x0/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.160364] [] ? child_rip+0x0/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162529] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162531] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162573] CPU 6 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162574] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162610] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162612] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162614] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162619] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162620] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162622] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162624] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162625] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162627] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162629] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162631] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162632] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162635] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162637] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162639] Stack: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162639] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162642] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162644] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162647] Call Trace: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162682] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162729] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162779] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162825] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162872] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162888] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162899] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162910] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162922] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162973] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162983] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.162999] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163030] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163047] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163075] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163090] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163105] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163156] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163160] [] ? strlcpy+0x4a/0x60 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163176] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163191] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163205] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163220] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163264] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163304] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163359] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163413] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163466] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163518] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163522] [] ? thread_return+0x4e/0x7d0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163574] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163577] [] ? kthread+0x9e/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163580] [] ? child_rip+0xa/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163583] [] ? kthread+0x0/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163585] [] ? child_rip+0x0/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163586] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163601] Call Trace: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163644] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163690] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163740] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163786] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163833] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163848] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163859] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163871] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163883] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163933] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163943] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163959] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.163987] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164004] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164032] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164047] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164062] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164113] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164117] [] ? strlcpy+0x4a/0x60 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164133] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164147] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164162] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164177] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164221] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164261] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164315] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164369] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164423] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164474] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164478] [] ? thread_return+0x4e/0x7d0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164530] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164533] [] ? kthread+0x9e/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164536] [] ? child_rip+0xa/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164538] [] ? kthread+0x0/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.164540] [] ? child_rip+0x0/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165528] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165530] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165570] CPU 7 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165571] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165606] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165608] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165611] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165615] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165617] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165618] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165620] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165622] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165623] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165625] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165627] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165629] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165632] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165634] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165636] Stack: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165637] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165639] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165641] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165643] Call Trace: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165677] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165724] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165774] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165820] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165867] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165883] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165893] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165905] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165916] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165931] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165947] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165976] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.165991] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166006] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166056] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166060] [] ? strlcpy+0x4a/0x60 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166076] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166091] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166106] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166120] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166164] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166205] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166259] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166312] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166366] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166418] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166421] [] ? thread_return+0x4e/0x7d0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166473] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166477] [] ? kthread+0x9e/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166480] [] ? child_rip+0xa/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166482] [] ? kthread+0x0/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166484] [] ? child_rip+0x0/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166486] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166501] Call Trace: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166543] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166590] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166639] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166686] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166732] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166748] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166759] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166770] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166781] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166796] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166812] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166841] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166855] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166870] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166920] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166924] [] ? strlcpy+0x4a/0x60 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166940] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166955] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166969] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.166984] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167028] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167068] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167122] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167176] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167230] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167281] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167285] [] ? thread_return+0x4e/0x7d0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167337] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167340] [] ? kthread+0x9e/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167343] [] ? child_rip+0xa/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167346] [] ? kthread+0x0/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484266.167348] [] ? child_rip+0x0/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.690271] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.787844] CPU 4 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.789960] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.887627] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.889509] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.902950] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.911670] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.917563] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.925468] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.933442] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.941414] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.949336] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.957215] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.966150] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.972469] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.980390] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.988355] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484268.996251] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.005723] Stack: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.008116] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.016159] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.024712] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.033492] Call Trace: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.036412] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.043799] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.051835] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.059495] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.066967] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.074579] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.082209] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.089640] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.096754] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.103683] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.111458] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.118665] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.126971] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.133313] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.140388] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.148366] [] ? strlcpy+0x4a/0x60 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.154004] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.160815] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.168232] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.175419] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.182683] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.190282] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.198392] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.205791] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.212708] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.220585] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.227704] [] ? thread_return+0x4e/0x7d0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.234002] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.240887] [] ? kthread+0x9e/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.246529] [] ? child_rip+0xa/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.252231] [] ? kthread+0x0/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.257778] [] ? child_rip+0x0/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.263453] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.284904] Call Trace: Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.287841] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.295280] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.303352] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.311020] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.321255] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.328825] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.336422] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.343764] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.350808] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.357729] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.365600] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.372796] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.381164] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.387518] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.394538] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.402552] [] ? strlcpy+0x4a/0x60 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.408233] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.415042] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.422402] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.429603] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.436842] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.444407] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.452429] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.459803] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.466685] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.474519] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.481543] [] ? thread_return+0x4e/0x7d0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.487857] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.494699] [] ? kthread+0x9e/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.500289] [] ? child_rip+0xa/0x20 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.505948] [] ? kthread+0x0/0xc0 Aug 25 20:33:27 feral28.ccs.ornl.gov kernel: [1484269.511416] [] ? child_rip+0x0/0x20 Aug 25 20:34:02 feral28.ccs.ornl.gov kernel: [1484304.949036] Lustre: sultan-MDT000b: haven't heard from client 2d302f5b-b294-591e-d666-ff70376d37f3 (at 10@gni1) in 238 seconds. I think it's dead, and I am evicting it. exp ffff880ba7992c00, cur 1472171642 expire 1472171420 last 1472171404 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.162138] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165138] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165140] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165182] CPU 5 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165183] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165219] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165221] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165224] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165228] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165230] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165231] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165233] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165234] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165236] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165238] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165240] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165242] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165244] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165245] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165247] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165249] Stack: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165250] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165252] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165254] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165256] Call Trace: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165300] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165346] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165396] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165443] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165489] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165505] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165516] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165528] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165538] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165553] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165570] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165599] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165613] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165628] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165679] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165683] [] ? strlcpy+0x4a/0x60 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165699] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165714] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165729] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165744] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165788] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165828] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165882] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165936] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.165990] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166042] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166045] [] ? thread_return+0x4e/0x7d0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166097] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166100] [] ? kthread+0x9e/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166103] [] ? child_rip+0xa/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166106] [] ? kthread+0x0/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166108] [] ? child_rip+0x0/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166109] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166124] Call Trace: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166167] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166213] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166263] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166309] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166356] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166372] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166382] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166394] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166405] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166420] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166436] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166464] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166479] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166494] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166544] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166548] [] ? strlcpy+0x4a/0x60 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166564] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166579] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166594] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166608] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166652] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166692] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166746] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166800] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166854] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166906] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166909] [] ? thread_return+0x4e/0x7d0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166961] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166965] [] ? kthread+0x9e/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166968] [] ? child_rip+0xa/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166971] [] ? kthread+0x0/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.166973] [] ? child_rip+0x0/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169138] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169140] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169183] CPU 6 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169183] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169219] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169221] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169224] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169228] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169230] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169231] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169233] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169234] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169236] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169238] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169240] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169241] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169245] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169247] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169248] Stack: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169249] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169251] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169253] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169256] Call Trace: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169291] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169337] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169387] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169434] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169481] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169497] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169507] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169519] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169531] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169582] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169592] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169608] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169637] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169653] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169682] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169697] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169712] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169763] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169767] [] ? strlcpy+0x4a/0x60 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169783] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169798] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169812] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169827] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169871] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169911] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.169965] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170019] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170073] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170125] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170128] [] ? thread_return+0x4e/0x7d0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170180] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170184] [] ? kthread+0x9e/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170186] [] ? child_rip+0xa/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170189] [] ? kthread+0x0/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170191] [] ? child_rip+0x0/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170193] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170208] Call Trace: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170251] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170297] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170347] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170393] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170440] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170455] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170466] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170478] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170490] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170540] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170550] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170566] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170595] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170612] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170640] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170655] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170670] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170720] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170724] [] ? strlcpy+0x4a/0x60 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170740] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170755] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170770] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170785] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170829] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170869] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170923] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.170977] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.171031] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.171083] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.171086] [] ? thread_return+0x4e/0x7d0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.171138] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.171141] [] ? kthread+0x9e/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.171144] [] ? child_rip+0xa/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.171147] [] ? kthread+0x0/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.171149] [] ? child_rip+0x0/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172137] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172139] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172178] CPU 7 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172179] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172215] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172216] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172219] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172223] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172225] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172226] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172228] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172230] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172231] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172233] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172235] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172237] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172238] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172240] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172242] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172244] Stack: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172244] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172247] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172249] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172251] Call Trace: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172285] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172332] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172381] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172428] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172475] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172490] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172501] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172513] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172523] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172539] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172555] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172584] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172598] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172613] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172664] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172668] [] ? strlcpy+0x4a/0x60 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172684] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172698] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172713] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172728] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172772] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172812] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172866] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172920] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.172973] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173025] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173029] [] ? thread_return+0x4e/0x7d0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173080] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173083] [] ? kthread+0x9e/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173086] [] ? child_rip+0xa/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173089] [] ? kthread+0x0/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173091] [] ? child_rip+0x0/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173092] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173108] Call Trace: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173150] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173197] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173246] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173293] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173339] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173355] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173366] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173378] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173388] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173403] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173419] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173448] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173462] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173477] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173528] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173531] [] ? strlcpy+0x4a/0x60 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173548] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173562] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173577] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173592] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173636] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173676] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173730] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173784] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173837] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173889] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173892] [] ? thread_return+0x4e/0x7d0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173944] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173948] [] ? kthread+0x9e/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173951] [] ? child_rip+0xa/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173953] [] ? kthread+0x0/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484350.173956] [] ? child_rip+0x0/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.695598] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.792858] CPU 4 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.794999] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.892355] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.894211] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.907748] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.916476] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.922377] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.930351] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.938282] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.946185] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.954133] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.962108] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.971034] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.977342] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.985239] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484352.993202] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.001204] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.010694] Stack: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.013069] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.021042] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.029579] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.038480] Call Trace: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.041417] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.048796] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.059555] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.067239] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.074771] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.082315] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.089945] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.097339] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.104454] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.111331] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.119131] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.126383] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.134714] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.141085] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.148210] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.156172] [] ? strlcpy+0x4a/0x60 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.161801] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.168645] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.176073] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.183284] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.190576] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.198173] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.206265] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.213638] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.220572] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.228502] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.235578] [] ? thread_return+0x4e/0x7d0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.241943] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.248786] [] ? kthread+0x9e/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.254385] [] ? child_rip+0xa/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.260053] [] ? kthread+0x0/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.265556] [] ? child_rip+0x0/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.271301] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.292803] Call Trace: Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.295688] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.303119] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.311166] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.318859] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.326285] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.333795] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.341418] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.348794] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.355909] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.362846] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.370683] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.377896] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.386230] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.392642] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.399698] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.407704] [] ? strlcpy+0x4a/0x60 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.413367] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.420144] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.427552] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.434610] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.441970] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.449583] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.457659] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.465059] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.472020] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.480000] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.487119] [] ? thread_return+0x4e/0x7d0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.493495] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.500415] [] ? kthread+0x9e/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.506100] [] ? child_rip+0xa/0x20 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.511863] [] ? kthread+0x0/0xc0 Aug 25 20:34:51 feral28.ccs.ornl.gov kernel: [1484353.517349] [] ? child_rip+0x0/0x20 Aug 25 20:36:03 feral28.ccs.ornl.gov kernel: [1484426.182325] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 20:36:03 feral28.ccs.ornl.gov kernel: [1484426.193432] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 33 previous similar messages Aug 25 20:36:03 feral28.ccs.ornl.gov kernel: [1484426.204580] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.101@o2ib1 (207): c: 63, oc: 0, rc: 63 Aug 25 20:36:03 feral28.ccs.ornl.gov kernel: [1484426.218062] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 33 previous similar messages Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.168748] BUG: soft lockup - CPU#4 stuck for 68s! [mdt01_002:21487] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171748] BUG: soft lockup - CPU#5 stuck for 68s! [mdt01_007:17158] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171749] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171792] CPU 5 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171792] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171829] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171830] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171833] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171838] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171839] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171841] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171842] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171844] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171845] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171847] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171849] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171851] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171854] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171856] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171857] Stack: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171858] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171860] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171863] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171865] Call Trace: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171909] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.171955] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172005] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172052] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172098] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172114] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172125] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172137] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172147] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172163] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172179] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172208] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172223] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172238] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172288] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172292] [] ? strlcpy+0x4a/0x60 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172308] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172323] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172338] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172352] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172396] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172437] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172491] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172545] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172599] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172651] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172654] [] ? thread_return+0x4e/0x7d0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172706] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172709] [] ? kthread+0x9e/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172712] [] ? child_rip+0xa/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172715] [] ? kthread+0x0/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172717] [] ? child_rip+0x0/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172718] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172733] Call Trace: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172776] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172822] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172872] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172918] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172965] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172980] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.172991] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173003] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173013] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173029] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173044] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173073] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173088] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173103] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173153] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173157] [] ? strlcpy+0x4a/0x60 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173173] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173188] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173202] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173217] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173261] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173301] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173355] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173410] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173463] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173515] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173519] [] ? thread_return+0x4e/0x7d0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173571] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173574] [] ? kthread+0x9e/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173577] [] ? child_rip+0xa/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173580] [] ? kthread+0x0/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.173582] [] ? child_rip+0x0/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175748] BUG: soft lockup - CPU#6 stuck for 68s! [mdt01_000:21485] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175750] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175792] CPU 6 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175793] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175830] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175831] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175834] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175839] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175840] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175842] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175843] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175845] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175846] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175848] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175850] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175852] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175854] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175855] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175857] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175859] Stack: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175860] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175862] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175864] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175867] Call Trace: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175902] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175948] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.175998] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176045] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176092] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176107] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176118] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176130] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176142] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176192] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176203] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176218] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176250] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176266] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176295] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176310] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176325] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176375] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176379] [] ? strlcpy+0x4a/0x60 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176395] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176410] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176425] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176440] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176483] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176524] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176578] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176632] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176685] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176737] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176741] [] ? thread_return+0x4e/0x7d0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176792] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176796] [] ? kthread+0x9e/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176799] [] ? child_rip+0xa/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176801] [] ? kthread+0x0/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176804] [] ? child_rip+0x0/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176805] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176820] Call Trace: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176863] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176909] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.176959] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177005] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177052] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177068] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177078] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177090] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177102] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177152] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177163] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177178] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177207] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177224] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177252] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177267] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177282] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177332] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177336] [] ? strlcpy+0x4a/0x60 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177352] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177367] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177382] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177397] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177440] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177481] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177535] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177589] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177642] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177694] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177698] [] ? thread_return+0x4e/0x7d0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177749] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177753] [] ? kthread+0x9e/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177756] [] ? child_rip+0xa/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177758] [] ? kthread+0x0/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.177761] [] ? child_rip+0x0/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178747] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178748] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178791] CPU 7 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178791] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178828] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178830] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178833] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178837] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178838] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178840] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178841] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178843] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178845] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178847] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178849] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178851] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178854] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178856] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178857] Stack: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178858] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178860] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178862] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178865] Call Trace: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178899] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178946] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.178996] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179042] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179089] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179105] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179115] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179127] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179138] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179153] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179169] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179198] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179213] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179228] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179279] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179283] [] ? strlcpy+0x4a/0x60 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179299] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179313] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179328] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179343] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179387] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179427] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179481] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179535] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179589] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179641] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179644] [] ? thread_return+0x4e/0x7d0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179696] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179699] [] ? kthread+0x9e/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179702] [] ? child_rip+0xa/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179705] [] ? kthread+0x0/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179707] [] ? child_rip+0x0/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179708] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179724] Call Trace: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179766] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179813] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179862] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179909] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179956] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179971] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179982] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.179994] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180004] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180019] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180036] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180064] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180079] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180094] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180144] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180148] [] ? strlcpy+0x4a/0x60 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180164] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180179] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180194] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180208] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180252] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180292] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180346] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180400] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180454] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180506] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180509] [] ? thread_return+0x4e/0x7d0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180561] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180564] [] ? kthread+0x9e/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180567] [] ? child_rip+0xa/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180570] [] ? kthread+0x0/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484434.180572] [] ? child_rip+0x0/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.682280] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.779376] CPU 4 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.781499] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.881473] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.883346] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.896796] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.905464] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.911323] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.919166] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.927000] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.934904] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.942800] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.950722] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.959631] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.965908] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.973767] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.981629] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.989524] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484436.998971] Stack: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.001381] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.009303] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.017857] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.026669] Call Trace: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.029574] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.036952] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.044973] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.052484] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.059919] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.067428] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.075058] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.082324] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.089411] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.096264] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.104030] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.111211] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.119438] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.125766] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.132724] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.140602] [] ? strlcpy+0x4a/0x60 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.146178] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.153066] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.160433] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.167566] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.174927] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.182507] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.190582] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.198018] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.204986] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.212897] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.220016] [] ? thread_return+0x4e/0x7d0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.226313] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.233225] [] ? kthread+0x9e/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.238856] [] ? child_rip+0xa/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.244508] [] ? kthread+0x0/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.250021] [] ? child_rip+0x0/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.255731] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.277199] Call Trace: Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.280099] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.287567] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.295569] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.303166] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.310603] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.318129] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.325717] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.333059] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.340087] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.346938] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.354679] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.361816] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.370078] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.376389] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.383366] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.391224] [] ? strlcpy+0x4a/0x60 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.396835] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.403707] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.411109] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.418354] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.425681] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.433278] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.441396] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.448892] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.455817] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.463692] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.470761] [] ? thread_return+0x4e/0x7d0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.477110] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.484064] [] ? kthread+0x9e/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.489610] [] ? child_rip+0xa/0x20 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.495270] [] ? kthread+0x0/0xc0 Aug 25 20:36:15 feral28.ccs.ornl.gov kernel: [1484437.500797] [] ? child_rip+0x0/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.175357] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178357] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178359] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178401] CPU 5 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178402] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178439] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178441] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178444] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178448] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178449] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178451] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178453] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178454] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178456] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178458] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178459] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178461] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178465] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178467] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178468] Stack: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178469] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178471] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178474] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178476] Call Trace: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178520] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178566] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178616] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178663] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178709] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178725] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178736] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178748] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178759] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178774] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178790] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178819] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178834] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178849] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178899] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178903] [] ? strlcpy+0x4a/0x60 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178920] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178934] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178949] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.178964] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179008] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179048] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179103] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179156] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179210] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179262] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179266] [] ? thread_return+0x4e/0x7d0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179317] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179321] [] ? kthread+0x9e/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179323] [] ? child_rip+0xa/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179326] [] ? kthread+0x0/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179328] [] ? child_rip+0x0/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179330] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179345] Call Trace: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179387] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179434] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179483] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179530] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179576] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179592] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179603] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179614] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179625] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179640] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179656] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179685] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179700] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179714] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179765] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179769] [] ? strlcpy+0x4a/0x60 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179785] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179799] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179814] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179829] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179873] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179913] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.179967] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.180021] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.180075] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.180126] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.180130] [] ? thread_return+0x4e/0x7d0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.180182] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.180185] [] ? kthread+0x9e/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.180188] [] ? child_rip+0xa/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.180190] [] ? kthread+0x0/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.180193] [] ? child_rip+0x0/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182357] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182359] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182402] CPU 6 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182403] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182440] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182442] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182445] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182449] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182451] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182452] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182454] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182455] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182457] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182459] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182461] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182463] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182465] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182466] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182469] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182470] Stack: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182471] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182473] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182475] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182477] Call Trace: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182512] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182559] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182609] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182656] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182702] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182718] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182729] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182741] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182753] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182803] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182814] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182830] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182859] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182875] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182904] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182919] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182934] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182984] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.182988] [] ? strlcpy+0x4a/0x60 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183004] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183019] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183034] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183049] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183092] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183133] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183187] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183241] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183295] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183346] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183350] [] ? thread_return+0x4e/0x7d0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183402] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183405] [] ? kthread+0x9e/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183408] [] ? child_rip+0xa/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183411] [] ? kthread+0x0/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183413] [] ? child_rip+0x0/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183414] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183429] Call Trace: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183472] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183519] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183568] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183615] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183661] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183677] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183687] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183699] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183711] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183761] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183772] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183787] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183816] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183832] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183861] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183875] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183890] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183941] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183945] [] ? strlcpy+0x4a/0x60 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183961] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183975] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.183990] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184005] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184049] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184089] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184143] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184198] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184251] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184303] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184307] [] ? thread_return+0x4e/0x7d0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184358] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184362] [] ? kthread+0x9e/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184365] [] ? child_rip+0xa/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184368] [] ? kthread+0x0/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.184370] [] ? child_rip+0x0/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185356] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185358] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185398] CPU 7 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185399] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185434] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185436] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185439] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185443] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185445] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185446] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185448] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185449] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185451] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185453] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185455] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185456] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185460] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185462] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185463] Stack: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185464] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185466] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185469] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185471] Call Trace: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185505] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185551] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185601] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185648] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185694] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185710] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185720] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185733] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185743] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185758] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185774] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185803] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185818] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185833] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185883] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185887] [] ? strlcpy+0x4a/0x60 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185903] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185918] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185933] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185948] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.185992] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186032] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186086] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186140] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186193] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186245] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186248] [] ? thread_return+0x4e/0x7d0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186300] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186303] [] ? kthread+0x9e/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186306] [] ? child_rip+0xa/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186309] [] ? kthread+0x0/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186311] [] ? child_rip+0x0/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186312] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186328] Call Trace: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186370] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186417] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186466] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186513] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186559] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186575] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186585] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186597] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186608] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186623] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186639] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186668] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186682] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186697] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186747] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186751] [] ? strlcpy+0x4a/0x60 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186767] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186782] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186797] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186812] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186855] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186896] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.186950] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.187003] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.187057] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.187109] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.187112] [] ? thread_return+0x4e/0x7d0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.187164] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.187167] [] ? kthread+0x9e/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.187170] [] ? child_rip+0xa/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.187172] [] ? kthread+0x0/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484518.187175] [] ? child_rip+0x0/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.710676] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.807903] CPU 4 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.810019] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.907574] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.909429] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.922905] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.931607] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.937544] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.945466] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.953431] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.961379] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.969327] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.977274] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.986227] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484520.992630] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.000579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.008544] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.016509] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.025930] Stack: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.028331] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.036278] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.044781] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.053595] Call Trace: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.056489] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.063885] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.071948] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.079615] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.087121] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.094698] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.102312] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.109672] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.116830] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.123733] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.131613] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.138817] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.147177] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.153591] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.160603] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.168556] [] ? strlcpy+0x4a/0x60 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.174229] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.181065] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.188466] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.195693] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.203004] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.210628] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.218762] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.226234] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.233175] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.241052] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.248153] [] ? thread_return+0x4e/0x7d0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.254494] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.261380] [] ? kthread+0x9e/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.266995] [] ? child_rip+0xa/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.272690] [] ? kthread+0x0/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.278209] [] ? child_rip+0x0/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.283868] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.305310] Call Trace: Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.308229] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.315684] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.323793] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.331495] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.338966] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.346525] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.354193] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.361587] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.368699] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.375648] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.383509] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.390776] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.399125] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.405531] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.412605] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.420540] [] ? strlcpy+0x4a/0x60 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.426177] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.433014] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.440346] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.447592] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.454910] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.462447] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.470579] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.478009] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.485011] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.492905] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.500024] [] ? thread_return+0x4e/0x7d0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.506381] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.513301] [] ? kthread+0x9e/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.518935] [] ? child_rip+0xa/0x20 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.524647] [] ? kthread+0x0/0xc0 Aug 25 20:37:39 feral28.ccs.ornl.gov kernel: [1484521.530217] [] ? child_rip+0x0/0x20 Aug 25 20:38:01 feral28.ccs.ornl.gov kernel: [1484543.407162] Lustre: sultan-MDT000b: haven't heard from client 8d48f2bb-3a94-5dec-2c5c-1f0ea8b2f1ab (at 25@gni1) in 315 seconds. I think it's dead, and I am evicting it. exp ffff880b974e4800, cur 1472171881 expire 1472171659 last 1472171566 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.181967] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.184967] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.184968] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185011] CPU 5 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185012] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185048] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185050] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185053] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185057] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185059] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185061] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185062] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185064] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185065] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185067] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185069] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185071] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185075] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185077] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185078] Stack: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185079] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185081] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185083] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185086] Call Trace: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185130] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185176] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185226] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185272] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185319] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185335] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185345] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185358] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185368] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185383] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185400] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185429] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185443] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185458] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185509] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185513] [] ? strlcpy+0x4a/0x60 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185529] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185544] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185559] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185573] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185617] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185658] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185712] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185766] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185820] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185871] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185875] [] ? thread_return+0x4e/0x7d0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185927] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185930] [] ? kthread+0x9e/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185933] [] ? child_rip+0xa/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185936] [] ? kthread+0x0/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185938] [] ? child_rip+0x0/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185939] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185954] Call Trace: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.185997] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186043] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186093] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186139] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186186] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186201] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186212] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186224] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186235] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186250] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186266] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186295] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186309] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186324] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186375] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186378] [] ? strlcpy+0x4a/0x60 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186395] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186409] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186424] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186439] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186483] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186523] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186577] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186631] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186685] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186737] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186740] [] ? thread_return+0x4e/0x7d0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186792] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186795] [] ? kthread+0x9e/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186798] [] ? child_rip+0xa/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186801] [] ? kthread+0x0/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.186803] [] ? child_rip+0x0/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.188967] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.188968] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189011] CPU 6 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189012] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189048] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189050] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189053] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189058] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189059] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189061] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189062] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189064] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189065] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189067] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189069] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189071] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189072] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189074] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189076] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189078] Stack: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189079] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189081] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189083] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189086] Call Trace: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189121] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189167] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189217] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189264] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189311] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189326] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189337] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189349] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189361] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189412] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189422] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189438] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189469] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189486] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189514] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189529] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189544] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189594] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189598] [] ? strlcpy+0x4a/0x60 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189614] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189629] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189644] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189658] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189702] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189743] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189797] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189851] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189904] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189956] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.189959] [] ? thread_return+0x4e/0x7d0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190011] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190014] [] ? kthread+0x9e/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190017] [] ? child_rip+0xa/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190020] [] ? kthread+0x0/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190022] [] ? child_rip+0x0/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190023] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190039] Call Trace: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190081] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190128] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190178] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190224] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190271] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190286] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190297] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190309] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190321] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190371] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190381] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190396] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190425] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190442] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190470] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190485] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190500] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190550] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190554] [] ? strlcpy+0x4a/0x60 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190570] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190585] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190600] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190615] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190658] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190699] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190753] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190807] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190860] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190912] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190916] [] ? thread_return+0x4e/0x7d0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190967] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190971] [] ? kthread+0x9e/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190974] [] ? child_rip+0xa/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190976] [] ? kthread+0x0/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.190978] [] ? child_rip+0x0/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.191966] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.191967] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192007] CPU 7 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192008] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192043] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192045] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192048] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192052] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192053] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192055] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192056] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192058] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192059] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192061] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192063] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192065] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192067] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192068] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192070] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192072] Stack: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192072] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192075] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192077] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192080] Call Trace: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192114] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192160] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192210] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192256] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192303] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192319] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192329] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192341] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192352] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192367] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192383] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192412] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192427] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192442] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192492] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192496] [] ? strlcpy+0x4a/0x60 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192512] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192527] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192542] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192557] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192601] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192641] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192695] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192749] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192803] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192854] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192858] [] ? thread_return+0x4e/0x7d0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192909] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192913] [] ? kthread+0x9e/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192916] [] ? child_rip+0xa/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192919] [] ? kthread+0x0/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192921] [] ? child_rip+0x0/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192922] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192937] Call Trace: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.192980] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193026] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193076] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193122] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193169] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193185] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193195] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193207] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193217] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193233] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193249] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193278] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193292] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193307] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193357] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193361] [] ? strlcpy+0x4a/0x60 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193377] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193392] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193407] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193421] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193465] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193506] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193560] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193614] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193667] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193719] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193722] [] ? thread_return+0x4e/0x7d0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193774] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193777] [] ? kthread+0x9e/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193780] [] ? child_rip+0xa/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193783] [] ? kthread+0x0/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484602.193785] [] ? child_rip+0x0/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.698791] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.795826] CPU 4 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.797898] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.895436] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.897317] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.910785] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.919452] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.925301] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.933197] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.941014] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.948901] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.956805] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.964658] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.973517] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.979876] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.987766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484604.995650] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.003520] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.012951] Stack: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.015324] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.023273] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.031784] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.040616] Call Trace: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.043527] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.050921] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.058907] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.066532] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.073960] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.081502] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.089065] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.096364] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.103383] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.110226] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.118018] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.125147] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.133401] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.139667] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.146671] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.154546] [] ? strlcpy+0x4a/0x60 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.160200] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.167003] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.174361] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.181572] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.188916] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.196574] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.204700] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.212180] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.219199] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.227180] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.234230] [] ? thread_return+0x4e/0x7d0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.240536] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.247412] [] ? kthread+0x9e/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.253011] [] ? child_rip+0xa/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.258679] [] ? kthread+0x0/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.264181] [] ? child_rip+0x0/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.269848] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.291369] Call Trace: Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.294295] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.301667] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.309636] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.317251] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.324696] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.332203] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.339766] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.347082] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.356822] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.363726] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.371526] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.378682] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.386936] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.393298] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.400327] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.408176] [] ? strlcpy+0x4a/0x60 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.413779] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.420659] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.428017] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.435220] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.442494] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.450109] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.458217] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.465601] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.472552] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.480429] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.487574] [] ? thread_return+0x4e/0x7d0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.493923] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.500843] [] ? kthread+0x9e/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.506416] [] ? child_rip+0xa/0x20 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.512118] [] ? kthread+0x0/0xc0 Aug 25 20:39:03 feral28.ccs.ornl.gov kernel: [1484605.517621] [] ? child_rip+0x0/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.188576] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191576] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191578] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191620] CPU 5 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191620] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191657] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191659] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191662] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191666] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191668] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191670] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191672] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191673] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191675] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191677] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191679] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191681] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191682] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191684] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191686] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191687] Stack: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191688] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191690] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191692] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191695] Call Trace: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191738] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191785] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191835] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191881] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191928] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191944] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191954] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191966] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191977] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.191992] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192008] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192038] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192052] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192067] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192117] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192121] [] ? strlcpy+0x4a/0x60 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192138] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192152] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192167] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192182] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192226] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192266] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192320] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192375] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192428] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192480] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192484] [] ? thread_return+0x4e/0x7d0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192535] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192539] [] ? kthread+0x9e/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192542] [] ? child_rip+0xa/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192545] [] ? kthread+0x0/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192547] [] ? child_rip+0x0/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192548] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192563] Call Trace: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192606] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192652] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192702] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192748] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192795] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192810] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192821] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192832] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192843] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192858] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192874] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192903] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192917] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192932] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192982] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.192986] [] ? strlcpy+0x4a/0x60 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193003] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193017] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193032] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193047] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193091] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193131] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193185] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193239] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193293] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193345] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193348] [] ? thread_return+0x4e/0x7d0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193400] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193404] [] ? kthread+0x9e/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193406] [] ? child_rip+0xa/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193409] [] ? kthread+0x0/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.193411] [] ? child_rip+0x0/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195576] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195578] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195621] CPU 6 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195622] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195659] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195661] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195663] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195668] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195669] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195671] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195673] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195674] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195676] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195678] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195680] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195682] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195683] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195685] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195687] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195689] Stack: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195690] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195692] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195694] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195696] Call Trace: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195731] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195778] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195828] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195874] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195921] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195937] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195948] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195959] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.195971] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196021] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196032] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196048] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196077] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196093] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196122] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196136] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196152] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196202] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196206] [] ? strlcpy+0x4a/0x60 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196222] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196237] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196251] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196266] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196310] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196350] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196404] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196458] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196512] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196564] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196567] [] ? thread_return+0x4e/0x7d0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196619] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196622] [] ? kthread+0x9e/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196625] [] ? child_rip+0xa/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196628] [] ? kthread+0x0/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196630] [] ? child_rip+0x0/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196631] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196647] Call Trace: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196689] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196736] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196786] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196832] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196879] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196894] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196905] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196917] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196929] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196979] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.196989] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197005] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197034] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197050] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197079] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197094] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197109] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197160] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197163] [] ? strlcpy+0x4a/0x60 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197180] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197194] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197209] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197224] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197268] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197308] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197362] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197416] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197470] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197522] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197525] [] ? thread_return+0x4e/0x7d0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197577] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197581] [] ? kthread+0x9e/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197583] [] ? child_rip+0xa/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197586] [] ? kthread+0x0/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.197588] [] ? child_rip+0x0/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198576] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198577] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198617] CPU 7 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198618] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198654] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198656] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198658] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198662] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198664] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198666] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198667] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198669] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198670] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198672] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198674] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198675] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198679] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198681] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198682] Stack: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198683] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198685] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198687] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198690] Call Trace: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198724] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198770] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198820] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198867] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198913] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198929] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198939] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198951] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198962] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198977] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.198993] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199022] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199037] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199052] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199102] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199106] [] ? strlcpy+0x4a/0x60 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199122] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199137] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199152] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199167] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199211] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199251] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199305] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199359] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199413] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199464] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199468] [] ? thread_return+0x4e/0x7d0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199520] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199523] [] ? kthread+0x9e/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199526] [] ? child_rip+0xa/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199529] [] ? kthread+0x0/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199531] [] ? child_rip+0x0/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199532] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199547] Call Trace: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199590] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199636] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199686] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199732] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199779] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199795] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199805] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199817] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199827] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199843] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199859] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199887] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199902] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199917] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199967] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199971] [] ? strlcpy+0x4a/0x60 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.199987] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200002] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200017] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200032] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200076] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200116] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200169] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200223] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200277] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200329] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200332] [] ? thread_return+0x4e/0x7d0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200384] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200387] [] ? kthread+0x9e/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200390] [] ? child_rip+0xa/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200393] [] ? kthread+0x0/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484686.200395] [] ? child_rip+0x0/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.721762] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.819231] CPU 4 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.821354] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.919092] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.920956] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.934380] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.943065] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.948959] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.956888] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.964854] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.972767] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.980749] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.988698] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484688.997675] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.004012] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.011983] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.019889] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.027890] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.037370] Stack: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.039805] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.047752] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.056238] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.065078] Call Trace: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.068007] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.075437] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.083483] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.091175] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.098700] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.106286] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.113952] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.121277] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.128382] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.135304] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.145852] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.153076] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.161408] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.167822] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.174886] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.182779] [] ? strlcpy+0x4a/0x60 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.188433] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.195260] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.202672] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.209822] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.217071] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.224659] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.232751] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.240239] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.247268] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.255177] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.262341] [] ? thread_return+0x4e/0x7d0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.268716] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.275619] [] ? kthread+0x9e/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.281235] [] ? child_rip+0xa/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.286946] [] ? kthread+0x0/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.292482] [] ? child_rip+0x0/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.298203] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.319600] Call Trace: Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.322495] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.329952] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.338024] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.345743] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.353257] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.360851] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.368499] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.375827] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.382923] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.389896] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.397731] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.404912] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.413244] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.419587] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.426610] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.434578] [] ? strlcpy+0x4a/0x60 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.440157] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.446994] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.454412] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.461649] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.468924] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.476556] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.484630] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.492005] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.498956] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.506824] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.513857] [] ? thread_return+0x4e/0x7d0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.520068] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.526900] [] ? kthread+0x9e/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.532473] [] ? child_rip+0xa/0x20 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.538139] [] ? kthread+0x0/0xc0 Aug 25 20:40:27 feral28.ccs.ornl.gov kernel: [1484689.543652] [] ? child_rip+0x0/0x20 Aug 25 20:40:58 feral28.ccs.ornl.gov kernel: [1484720.624562] Lustre: sultan-MDT000b: haven't heard from client cc3b2647-4e02-c39c-f43e-826f0fc767f8 (at 28@gni1) in 337 seconds. I think it's dead, and I am evicting it. exp ffff880b9dd49c00, cur 1472172058 expire 1472171836 last 1472171721 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.195185] BUG: soft lockup - CPU#4 stuck for 68s! [mdt01_002:21487] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198185] BUG: soft lockup - CPU#5 stuck for 68s! [mdt01_007:17158] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198186] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198228] CPU 5 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198229] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198265] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198267] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198270] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198275] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198276] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198278] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198280] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198281] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198283] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198285] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198287] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198289] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198290] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198292] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198294] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198295] Stack: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198296] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198298] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198300] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198303] Call Trace: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198346] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198393] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198443] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198489] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198536] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198552] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198562] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198574] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198585] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198600] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198616] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198645] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198660] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198675] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198725] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198729] [] ? strlcpy+0x4a/0x60 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198746] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198760] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198775] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198790] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198834] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198874] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198929] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.198983] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199036] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199088] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199092] [] ? thread_return+0x4e/0x7d0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199143] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199147] [] ? kthread+0x9e/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199150] [] ? child_rip+0xa/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199152] [] ? kthread+0x0/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199154] [] ? child_rip+0x0/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199156] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199171] Call Trace: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199213] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199260] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199310] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199356] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199403] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199418] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199429] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199441] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199451] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199466] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199482] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199511] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199526] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199541] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199591] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199595] [] ? strlcpy+0x4a/0x60 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199611] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199626] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199641] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199655] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199699] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199740] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199794] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199848] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199901] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199953] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.199957] [] ? thread_return+0x4e/0x7d0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.200008] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.200012] [] ? kthread+0x9e/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.200015] [] ? child_rip+0xa/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.200017] [] ? kthread+0x0/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.200020] [] ? child_rip+0x0/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202185] BUG: soft lockup - CPU#6 stuck for 68s! [mdt01_000:21485] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202187] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202229] CPU 6 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202230] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202267] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202268] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202271] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202276] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202277] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202279] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202280] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202282] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202283] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202285] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202287] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202289] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202291] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202292] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202294] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202296] Stack: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202297] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202299] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202301] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202304] Call Trace: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202339] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202385] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202435] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202482] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202529] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202544] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202555] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202567] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202579] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202629] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202640] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202655] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202686] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202703] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202732] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202746] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202761] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202812] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202816] [] ? strlcpy+0x4a/0x60 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202832] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202847] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202862] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202877] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202920] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.202961] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203015] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203069] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203122] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203174] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203178] [] ? thread_return+0x4e/0x7d0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203229] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203233] [] ? kthread+0x9e/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203236] [] ? child_rip+0xa/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203239] [] ? kthread+0x0/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203241] [] ? child_rip+0x0/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203242] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203257] Call Trace: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203300] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203347] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203396] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203443] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203489] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203505] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203515] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203527] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203539] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203589] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203600] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203615] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203646] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203662] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203691] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203706] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203721] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203771] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203775] [] ? strlcpy+0x4a/0x60 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203791] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203806] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203821] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203835] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203879] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203919] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.203974] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.204028] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.204082] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.204134] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.204137] [] ? thread_return+0x4e/0x7d0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.204189] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.204193] [] ? kthread+0x9e/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.204195] [] ? child_rip+0xa/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.204198] [] ? kthread+0x0/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.204200] [] ? child_rip+0x0/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205184] BUG: soft lockup - CPU#7 stuck for 68s! [mdt01_001:21486] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205185] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205225] CPU 7 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205226] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205261] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205263] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205266] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205270] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205272] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205273] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205275] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205276] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205278] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205280] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205282] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205283] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205285] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205286] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205288] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205290] Stack: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205291] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205293] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205295] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205298] Call Trace: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205331] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205378] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205428] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205474] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205521] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205536] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205547] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205559] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205570] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205585] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205601] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205630] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205645] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205659] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205710] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205714] [] ? strlcpy+0x4a/0x60 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205730] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205745] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205760] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205774] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205818] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205859] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205913] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.205967] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206020] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206072] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206075] [] ? thread_return+0x4e/0x7d0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206127] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206130] [] ? kthread+0x9e/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206133] [] ? child_rip+0xa/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206136] [] ? kthread+0x0/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206138] [] ? child_rip+0x0/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206139] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206155] Call Trace: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206197] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206244] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206293] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206340] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206387] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206402] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206413] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206424] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206435] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206450] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206466] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206494] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206509] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206524] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206574] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206578] [] ? strlcpy+0x4a/0x60 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206594] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206609] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206624] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206639] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206682] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206723] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206777] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206831] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206884] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206936] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206939] [] ? thread_return+0x4e/0x7d0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206991] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206994] [] ? kthread+0x9e/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.206997] [] ? child_rip+0xa/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.207000] [] ? kthread+0x0/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484770.207002] [] ? child_rip+0x0/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.727807] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.825137] CPU 4 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.827252] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.924745] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.926568] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.942705] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.951425] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.957326] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.965265] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.973185] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.981109] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.989004] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484772.996961] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.005896] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.012241] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.020212] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.028162] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.036153] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.045651] Stack: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.048078] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.056043] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.064546] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.073403] Call Trace: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.076296] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.083719] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.091765] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.099483] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.107041] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.114592] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.122250] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.129567] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.136662] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.143540] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.151386] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.158628] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.166960] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.173286] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.180334] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.188295] [] ? strlcpy+0x4a/0x60 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.193924] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.200769] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.208101] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.215330] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.222648] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.230226] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.238250] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.245677] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.252706] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.260574] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.267675] [] ? thread_return+0x4e/0x7d0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.273973] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.280840] [] ? kthread+0x9e/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.286448] [] ? child_rip+0xa/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.292194] [] ? kthread+0x0/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.297722] [] ? child_rip+0x0/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.303434] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.324857] Call Trace: Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.327794] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.335260] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.343350] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.351006] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.358549] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.366092] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.373738] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.381169] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.388284] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.395194] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.403057] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.410246] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.418587] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.424992] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.432064] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.439993] [] ? strlcpy+0x4a/0x60 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.445639] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.452450] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.459825] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.467002] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.474344] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.481951] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.489991] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.497427] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.504376] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.512314] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.519373] [] ? thread_return+0x4e/0x7d0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.525653] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.532573] [] ? kthread+0x9e/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.538267] [] ? child_rip+0xa/0x20 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.544004] [] ? kthread+0x0/0xc0 Aug 25 20:41:51 feral28.ccs.ornl.gov kernel: [1484773.549507] [] ? child_rip+0x0/0x20 Aug 25 20:42:24 feral28.ccs.ornl.gov kernel: [1484806.903695] Lustre: sultan-MDT000b: Client ade025d5-35bd-119e-3200-b4438b686c43 (at 20@gni1) reconnecting Aug 25 20:42:24 feral28.ccs.ornl.gov kernel: [1484806.914161] Lustre: Skipped 9 previous similar messages Aug 25 20:42:24 feral28.ccs.ornl.gov kernel: [1484806.919969] Lustre: sultan-MDT000b: Connection restored to 84595275-af73-a989-8d49-91a86f054b0f (at 20@gni1) Aug 25 20:42:24 feral28.ccs.ornl.gov kernel: [1484806.930622] Lustre: Skipped 12 previous similar messages Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.201788] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204788] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204790] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204832] CPU 5 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204833] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204869] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204871] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204874] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204878] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204880] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204881] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204883] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204885] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204887] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204889] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204891] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204892] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204896] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204898] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204899] Stack: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204900] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204902] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204904] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204906] Call Trace: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204950] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.204997] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205046] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205093] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205140] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205155] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205166] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205178] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205188] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205204] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205220] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205249] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205264] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205279] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205329] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205333] [] ? strlcpy+0x4a/0x60 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205349] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205364] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205379] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205394] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205437] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205478] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205532] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205586] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205640] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205691] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205695] [] ? thread_return+0x4e/0x7d0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205747] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205750] [] ? kthread+0x9e/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205753] [] ? child_rip+0xa/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205756] [] ? kthread+0x0/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205758] [] ? child_rip+0x0/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205759] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205774] Call Trace: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205817] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205863] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205913] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.205959] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206006] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206021] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206032] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206044] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206055] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206070] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206086] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206114] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206129] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206144] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206194] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206198] [] ? strlcpy+0x4a/0x60 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206214] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206229] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206244] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206258] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206302] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206343] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206397] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206451] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206505] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206556] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206560] [] ? thread_return+0x4e/0x7d0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206611] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206615] [] ? kthread+0x9e/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206618] [] ? child_rip+0xa/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206620] [] ? kthread+0x0/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.206623] [] ? child_rip+0x0/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208788] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208790] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208832] CPU 6 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208833] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208870] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208872] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208875] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208879] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208881] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208883] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208884] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208886] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208887] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208889] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208891] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208893] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208896] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208898] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208900] Stack: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208901] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208903] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208905] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208908] Call Trace: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208942] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.208989] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209039] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209085] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209132] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209148] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209158] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209170] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209182] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209232] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209243] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209258] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209288] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209304] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209333] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209348] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209363] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209413] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209418] [] ? strlcpy+0x4a/0x60 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209434] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209449] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209463] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209478] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209522] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209563] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209617] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209671] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209724] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209776] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209780] [] ? thread_return+0x4e/0x7d0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209831] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209835] [] ? kthread+0x9e/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209837] [] ? child_rip+0xa/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209840] [] ? kthread+0x0/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209842] [] ? child_rip+0x0/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209844] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209859] Call Trace: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209902] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209948] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.209998] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210044] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210091] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210106] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210117] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210129] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210141] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210191] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210202] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210217] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210246] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210263] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210291] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210306] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210321] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210372] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210376] [] ? strlcpy+0x4a/0x60 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210392] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210406] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210421] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210436] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210480] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210520] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210574] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210628] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210682] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210734] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210737] [] ? thread_return+0x4e/0x7d0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210789] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210792] [] ? kthread+0x9e/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210795] [] ? child_rip+0xa/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210798] [] ? kthread+0x0/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.210800] [] ? child_rip+0x0/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211787] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211789] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211828] CPU 7 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211829] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211865] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211866] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211869] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211873] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211875] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211876] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211878] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211879] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211881] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211883] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211885] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211886] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211888] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211890] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211892] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211893] Stack: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211894] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211896] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211898] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211901] Call Trace: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211935] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.211981] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212031] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212078] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212124] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212140] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212151] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212162] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212173] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212188] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212204] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212233] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212248] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212263] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212313] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212317] [] ? strlcpy+0x4a/0x60 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212333] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212348] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212363] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212378] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212422] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212462] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212516] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212570] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212624] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212676] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212679] [] ? thread_return+0x4e/0x7d0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212731] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212734] [] ? kthread+0x9e/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212737] [] ? child_rip+0xa/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212740] [] ? kthread+0x0/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212742] [] ? child_rip+0x0/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212743] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212758] Call Trace: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212801] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212847] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212897] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212944] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.212990] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213006] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213017] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213028] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213039] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213054] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213070] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213099] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213113] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213128] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213179] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213183] [] ? strlcpy+0x4a/0x60 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213199] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213214] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213229] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213243] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213287] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213328] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213381] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213436] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213489] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213541] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213544] [] ? thread_return+0x4e/0x7d0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213596] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213600] [] ? kthread+0x9e/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213602] [] ? child_rip+0xa/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213605] [] ? kthread+0x0/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484854.213607] [] ? child_rip+0x0/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.717800] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.814930] CPU 4 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.817019] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.914394] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.916259] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.929657] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.938288] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.944165] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.952062] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.959948] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.967844] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.975729] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.983573] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.992492] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484856.998767] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.006611] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.014454] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.022332] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.031666] Stack: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.034051] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.041996] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.050525] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.059349] Call Trace: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.062251] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.069648] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.077676] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.085308] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.092702] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.100176] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.107754] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.115046] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.122125] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.129009] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.136847] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.144034] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.152272] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.158591] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.165612] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.173531] [] ? strlcpy+0x4a/0x60 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.179202] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.186048] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.193363] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.200522] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.207822] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.215374] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.223416] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.230852] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.237820] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.245784] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.252885] [] ? thread_return+0x4e/0x7d0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.259234] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.266155] [] ? kthread+0x9e/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.271727] [] ? child_rip+0xa/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.277386] [] ? kthread+0x0/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.282904] [] ? child_rip+0x0/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.288616] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.310102] Call Trace: Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.313013] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.320392] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.328384] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.335958] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.343411] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.350904] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.358508] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.365841] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.372886] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.379753] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.387521] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.394711] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.402930] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.409231] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.416251] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.424161] [] ? strlcpy+0x4a/0x60 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.429806] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.436575] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.443951] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.451144] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.458419] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.466042] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.474082] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.481466] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.488434] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.496363] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.503474] [] ? thread_return+0x4e/0x7d0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.509823] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.516708] [] ? kthread+0x9e/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.522288] [] ? child_rip+0xa/0x20 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.527973] [] ? kthread+0x0/0xc0 Aug 25 20:43:15 feral28.ccs.ornl.gov kernel: [1484857.533495] [] ? child_rip+0x0/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.208391] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211391] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211393] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211436] CPU 5 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211437] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211473] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211475] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211478] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211482] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211484] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211486] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211487] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211489] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211491] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211493] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211495] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211496] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211500] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211502] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211503] Stack: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211504] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211506] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211508] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211510] Call Trace: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211554] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211600] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211650] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211697] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211744] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211759] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211770] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211782] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211793] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211808] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211824] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211853] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211868] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211883] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211933] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211937] [] ? strlcpy+0x4a/0x60 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211954] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211968] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211983] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.211998] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212042] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212082] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212136] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212190] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212244] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212296] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212299] [] ? thread_return+0x4e/0x7d0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212351] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212355] [] ? kthread+0x9e/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212357] [] ? child_rip+0xa/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212360] [] ? kthread+0x0/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212363] [] ? child_rip+0x0/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212364] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212379] Call Trace: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212422] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212468] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212518] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212564] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212611] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212626] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212637] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212649] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212659] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212674] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212690] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212719] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212733] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212748] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212798] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212802] [] ? strlcpy+0x4a/0x60 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212818] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212833] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212848] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212863] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212907] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.212947] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213002] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213056] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213109] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213161] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213165] [] ? thread_return+0x4e/0x7d0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213216] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213220] [] ? kthread+0x9e/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213223] [] ? child_rip+0xa/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213225] [] ? kthread+0x0/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.213228] [] ? child_rip+0x0/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215391] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215393] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215436] CPU 6 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215437] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215474] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215476] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215479] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215483] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215485] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215486] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215488] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215490] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215491] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215493] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215496] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215497] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215499] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215501] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215503] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215505] Stack: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215506] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215508] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215510] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215512] Call Trace: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215547] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215594] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215644] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215690] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215737] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215753] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215763] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215775] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215787] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215837] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215848] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215864] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215894] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215911] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215940] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215954] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.215969] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216020] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216024] [] ? strlcpy+0x4a/0x60 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216040] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216055] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216070] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216085] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216128] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216169] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216223] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216277] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216331] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216382] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216386] [] ? thread_return+0x4e/0x7d0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216438] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216441] [] ? kthread+0x9e/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216444] [] ? child_rip+0xa/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216447] [] ? kthread+0x0/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216449] [] ? child_rip+0x0/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216450] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216465] Call Trace: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216508] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216554] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216604] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216650] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216697] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216712] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216723] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216734] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216747] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216797] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216807] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216823] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216852] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216868] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216897] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216911] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216926] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216976] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216980] [] ? strlcpy+0x4a/0x60 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.216996] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217011] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217026] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217041] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217085] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217125] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217179] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217233] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217287] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217339] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217342] [] ? thread_return+0x4e/0x7d0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217394] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217397] [] ? kthread+0x9e/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217400] [] ? child_rip+0xa/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217403] [] ? kthread+0x0/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.217405] [] ? child_rip+0x0/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218391] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218392] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218432] CPU 7 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218433] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218468] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218470] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218473] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218477] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218479] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218480] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218482] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218483] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218485] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218487] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218489] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218490] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218492] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218493] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218495] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218497] Stack: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218497] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218499] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218501] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218504] Call Trace: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218537] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218584] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218634] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218680] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218727] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218742] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218753] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218765] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218775] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218790] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218807] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218835] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218850] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218865] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218915] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218919] [] ? strlcpy+0x4a/0x60 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218935] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218950] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218965] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.218980] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219024] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219064] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219118] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219172] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219225] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219277] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219280] [] ? thread_return+0x4e/0x7d0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219332] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219335] [] ? kthread+0x9e/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219338] [] ? child_rip+0xa/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219341] [] ? kthread+0x0/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219343] [] ? child_rip+0x0/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219344] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219359] Call Trace: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219402] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219449] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219498] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219545] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219592] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219607] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219618] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219630] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219640] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219655] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219671] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219700] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219715] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219730] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219780] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219784] [] ? strlcpy+0x4a/0x60 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219800] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219814] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219829] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219844] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219888] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219928] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.219982] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.220036] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.220090] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.220141] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.220145] [] ? thread_return+0x4e/0x7d0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.220197] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.220200] [] ? kthread+0x9e/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.220203] [] ? child_rip+0xa/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.220206] [] ? kthread+0x0/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484938.220208] [] ? child_rip+0x0/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.743573] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.840738] CPU 4 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.842878] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.940486] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.942377] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.955835] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.964442] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.970370] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.978298] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.986213] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484940.994100] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.002021] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.009969] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.018861] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.025258] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.033154] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.041143] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.049084] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.058590] Stack: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.061000] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.068896] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.077380] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.086204] Call Trace: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.089115] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.096476] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.104505] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.112207] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.119782] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.127325] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.134912] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.142230] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.149361] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.156290] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.164152] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.171359] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.179638] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.186010] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.193143] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.201105] [] ? strlcpy+0x4a/0x60 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.206759] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.213605] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.221023] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.228183] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.235518] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.243115] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.251137] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.258599] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.265515] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.273349] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.280442] [] ? thread_return+0x4e/0x7d0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.286694] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.293538] [] ? kthread+0x9e/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.299083] [] ? child_rip+0xa/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.304770] [] ? kthread+0x0/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.310281] [] ? child_rip+0x0/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.315957] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.337364] Call Trace: Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.340266] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.347662] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.355776] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.363471] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.370925] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.378468] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.386091] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.393537] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.400616] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.407545] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.415450] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.422614] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.430954] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.437367] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.444467] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.455099] [] ? strlcpy+0x4a/0x60 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.460736] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.467555] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.475009] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.482289] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.489572] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.497170] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.505270] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.512669] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.519613] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.527498] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.534601] [] ? thread_return+0x4e/0x7d0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.540959] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.547861] [] ? kthread+0x9e/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.553538] [] ? child_rip+0xa/0x20 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.559301] [] ? kthread+0x0/0xc0 Aug 25 20:44:39 feral28.ccs.ornl.gov kernel: [1484941.564787] [] ? child_rip+0x0/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.214995] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.217995] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.217996] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218038] CPU 5 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218039] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218075] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218077] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218080] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218084] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218085] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218087] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218089] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218090] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218092] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218094] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218096] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218098] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218099] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218103] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218104] Stack: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218105] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218107] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218109] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218112] Call Trace: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218156] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218202] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218252] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218299] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218345] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218361] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218372] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218384] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218394] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218409] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218426] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218455] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218470] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218484] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218535] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218539] [] ? strlcpy+0x4a/0x60 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218555] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218570] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218585] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218600] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218643] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218684] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218738] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218792] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218846] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218898] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218901] [] ? thread_return+0x4e/0x7d0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218953] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218956] [] ? kthread+0x9e/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218959] [] ? child_rip+0xa/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218962] [] ? kthread+0x0/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218964] [] ? child_rip+0x0/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218965] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.218980] Call Trace: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219023] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219069] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219119] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219165] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219212] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219228] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219238] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219250] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219260] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219275] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219292] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219320] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219335] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219350] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219400] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219404] [] ? strlcpy+0x4a/0x60 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219420] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219435] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219450] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219464] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219508] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219549] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219602] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219657] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219710] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219762] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219765] [] ? thread_return+0x4e/0x7d0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219817] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219821] [] ? kthread+0x9e/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219823] [] ? child_rip+0xa/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219826] [] ? kthread+0x0/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.219828] [] ? child_rip+0x0/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.221995] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.221997] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222039] CPU 6 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222040] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222077] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222079] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222083] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222087] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222089] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222090] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222092] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222094] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222095] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222097] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222099] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222101] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222103] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222104] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222106] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222108] Stack: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222108] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222110] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222113] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222115] Call Trace: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222150] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222197] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222246] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222293] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222340] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222355] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222366] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222378] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222390] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222440] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222451] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222466] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222495] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222512] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222540] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222555] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222570] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222620] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222624] [] ? strlcpy+0x4a/0x60 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222641] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222655] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222670] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222685] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222729] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222769] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222823] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222878] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222932] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222984] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.222987] [] ? thread_return+0x4e/0x7d0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223039] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223042] [] ? kthread+0x9e/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223045] [] ? child_rip+0xa/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223048] [] ? kthread+0x0/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223050] [] ? child_rip+0x0/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223051] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223067] Call Trace: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223109] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223155] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223205] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223251] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223298] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223313] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223324] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223336] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223348] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223398] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223408] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223424] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223453] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223469] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223498] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223512] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223527] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223578] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223581] [] ? strlcpy+0x4a/0x60 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223598] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223612] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223627] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223642] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223686] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223726] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223780] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223834] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223888] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223940] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223943] [] ? thread_return+0x4e/0x7d0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223995] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.223998] [] ? kthread+0x9e/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.224001] [] ? child_rip+0xa/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.224004] [] ? kthread+0x0/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.224006] [] ? child_rip+0x0/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.224994] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.224995] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225037] CPU 7 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225038] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225074] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225076] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225079] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225083] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225084] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225086] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225088] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225089] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225091] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225093] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225095] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225096] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225099] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225101] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225103] Stack: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225103] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225106] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225108] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225110] Call Trace: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225145] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225191] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225241] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225287] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225334] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225350] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225360] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225373] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225383] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225398] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225414] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225443] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225458] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225473] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225523] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225527] [] ? strlcpy+0x4a/0x60 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225543] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225558] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225573] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225588] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225632] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225672] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225726] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225780] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225834] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225886] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225889] [] ? thread_return+0x4e/0x7d0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225941] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225944] [] ? kthread+0x9e/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225947] [] ? child_rip+0xa/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225950] [] ? kthread+0x0/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225952] [] ? child_rip+0x0/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225953] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.225968] Call Trace: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226011] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226058] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226107] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226154] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226201] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226216] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226227] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226239] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226249] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226264] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226280] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226309] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226324] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226339] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226389] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226392] [] ? strlcpy+0x4a/0x60 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226409] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226423] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226438] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226453] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226497] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226537] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226591] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226645] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226699] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226751] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226754] [] ? thread_return+0x4e/0x7d0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226806] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226809] [] ? kthread+0x9e/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226812] [] ? child_rip+0xa/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226815] [] ? kthread+0x0/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485022.226817] [] ? child_rip+0x0/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.746948] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.844286] CPU 4 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.846375] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.944190] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.946038] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.959463] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.968207] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.974065] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.982014] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.989944] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485024.997805] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.005770] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.013735] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.022645] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.029007] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.036987] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.044953] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.052937] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.062452] Stack: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.064904] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.072843] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.081363] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.090264] Call Trace: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.093165] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.100580] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.108669] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.116336] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.123850] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.131421] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.139112] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.146522] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.153629] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.160524] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.168377] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.175601] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.183967] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.190338] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.197359] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.205347] [] ? strlcpy+0x4a/0x60 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.211036] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.220532] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.227892] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.235059] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.242342] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.249897] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.257954] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.265389] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.272314] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.280217] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.287327] [] ? thread_return+0x4e/0x7d0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.293660] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.300570] [] ? kthread+0x9e/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.306203] [] ? child_rip+0xa/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.311915] [] ? kthread+0x0/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.317435] [] ? child_rip+0x0/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.323155] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.344640] Call Trace: Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.347541] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.354982] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.363062] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.370685] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.378182] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.385768] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.393400] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.400786] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.407849] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.414753] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.422596] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.429830] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.438153] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.444541] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.451596] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.459566] [] ? strlcpy+0x4a/0x60 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.465177] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.471998] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.479424] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.486618] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.493936] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.501447] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.509545] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.516958] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.523847] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.531750] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.538860] [] ? thread_return+0x4e/0x7d0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.545175] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.552077] [] ? kthread+0x9e/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.557694] [] ? child_rip+0xa/0x20 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.563352] [] ? kthread+0x0/0xc0 Aug 25 20:46:03 feral28.ccs.ornl.gov kernel: [1485025.568925] [] ? child_rip+0x0/0x20 Aug 25 20:46:49 feral28.ccs.ornl.gov kernel: [1485072.234128] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds Aug 25 20:46:49 feral28.ccs.ornl.gov kernel: [1485072.245132] LNetError: 21204:0:(o2iblnd_cb.c:3113:kiblnd_check_txs_locked()) Skipped 33 previous similar messages Aug 25 20:46:49 feral28.ccs.ornl.gov kernel: [1485072.256283] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Timed out RDMA with 10.37.248.159@o2ib1 (225): c: 63, oc: 0, rc: 63 Aug 25 20:46:49 feral28.ccs.ornl.gov kernel: [1485072.269650] LNetError: 21204:0:(o2iblnd_cb.c:3176:kiblnd_check_conns()) Skipped 33 previous similar messages Aug 25 20:47:04 feral28.ccs.ornl.gov kernel: [1485086.725648] Lustre: sultan-MDT000b: haven't heard from client 121a88f7-96bf-42c6-fa63-63b545c0ecc1 (at 7@gni1) in 329 seconds. I think it's dead, and I am evicting it. exp ffff880ba1194000, cur 1472172424 expire 1472172202 last 1472172095 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.221598] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224598] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224600] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224642] CPU 5 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224643] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224679] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224680] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224683] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224688] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224689] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224691] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224693] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224694] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224696] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224698] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224700] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224702] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224705] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224707] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224708] Stack: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224709] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224711] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224713] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224715] Call Trace: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224759] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224806] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224855] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224902] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224949] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224965] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224975] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224987] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.224998] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225013] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225029] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225058] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225073] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225088] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225138] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225142] [] ? strlcpy+0x4a/0x60 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225159] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225173] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225188] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225203] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225247] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225287] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225341] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225395] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225449] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225500] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225504] [] ? thread_return+0x4e/0x7d0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225556] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225559] [] ? kthread+0x9e/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225562] [] ? child_rip+0xa/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225565] [] ? kthread+0x0/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225567] [] ? child_rip+0x0/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225568] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225583] Call Trace: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225626] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225672] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225722] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225768] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225815] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225831] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225841] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225853] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225864] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225879] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225895] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225924] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225938] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.225953] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226003] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226007] [] ? strlcpy+0x4a/0x60 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226023] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226038] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226053] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226068] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226112] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226152] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226206] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226260] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226314] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226366] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226369] [] ? thread_return+0x4e/0x7d0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226421] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226424] [] ? kthread+0x9e/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226427] [] ? child_rip+0xa/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226430] [] ? kthread+0x0/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.226432] [] ? child_rip+0x0/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228598] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228600] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228642] CPU 6 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228643] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228680] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228682] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228685] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228689] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228691] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228692] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228694] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228695] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228697] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228699] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228701] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228703] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228705] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228707] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228709] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228710] Stack: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228711] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228713] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228715] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228718] Call Trace: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228753] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228799] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228849] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228895] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228942] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228958] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228968] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228980] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.228992] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229043] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229053] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229069] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229100] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229116] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229145] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229160] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229175] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229225] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229229] [] ? strlcpy+0x4a/0x60 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229245] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229260] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229275] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229290] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229333] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229374] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229428] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229482] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229536] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229587] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229591] [] ? thread_return+0x4e/0x7d0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229643] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229646] [] ? kthread+0x9e/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229649] [] ? child_rip+0xa/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229652] [] ? kthread+0x0/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229654] [] ? child_rip+0x0/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229655] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229671] Call Trace: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229713] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229759] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229809] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229856] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229902] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229918] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229928] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229940] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.229952] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230002] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230013] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230028] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230057] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230073] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230102] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230116] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230131] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230182] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230186] [] ? strlcpy+0x4a/0x60 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230202] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230217] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230231] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230246] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230290] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230330] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230384] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230438] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230492] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230544] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230547] [] ? thread_return+0x4e/0x7d0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230599] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230602] [] ? kthread+0x9e/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230605] [] ? child_rip+0xa/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230608] [] ? kthread+0x0/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.230610] [] ? child_rip+0x0/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231597] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231599] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231638] CPU 7 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231639] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231675] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231677] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231679] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231684] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231685] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231687] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231688] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231690] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231691] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231693] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231695] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231697] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231700] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231702] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231703] Stack: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231704] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231706] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231708] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231710] Call Trace: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231744] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231790] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231840] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231887] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231933] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231949] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231960] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231972] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231983] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.231998] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232014] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232043] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232058] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232072] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232123] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232127] [] ? strlcpy+0x4a/0x60 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232143] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232158] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232172] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232187] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232231] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232271] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232325] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232379] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232432] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232484] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232487] [] ? thread_return+0x4e/0x7d0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232539] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232542] [] ? kthread+0x9e/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232545] [] ? child_rip+0xa/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232548] [] ? kthread+0x0/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232550] [] ? child_rip+0x0/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232552] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232567] Call Trace: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232609] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232656] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232705] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232752] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232798] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232814] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232824] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232836] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232847] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232862] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232878] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232907] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232921] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232936] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232987] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.232990] [] ? strlcpy+0x4a/0x60 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233007] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233021] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233036] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233051] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233095] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233135] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233189] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233243] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233297] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233348] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233352] [] ? thread_return+0x4e/0x7d0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233403] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233407] [] ? kthread+0x9e/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233410] [] ? child_rip+0xa/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233412] [] ? kthread+0x0/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485106.233415] [] ? child_rip+0x0/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.754389] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.851857] CPU 4 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.853998] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.951370] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.953236] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.966729] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.978039] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.983916] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.991785] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485108.999742] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.007707] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.015628] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.023637] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.032554] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.038934] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.046846] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.054761] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.062725] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.072189] Stack: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.074616] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.082562] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.091073] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.099862] Call Trace: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.102800] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.110265] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.118320] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.126030] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.133536] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.141123] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.148806] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.156139] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.163236] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.170165] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.177967] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.185164] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.193496] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.199798] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.206878] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.214807] [] ? strlcpy+0x4a/0x60 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.220478] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.227289] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.234690] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.241919] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.249237] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.256817] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.264872] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.272336] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.279294] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.287163] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.294168] [] ? thread_return+0x4e/0x7d0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.300458] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.307361] [] ? kthread+0x9e/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.312992] [] ? child_rip+0xa/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.318677] [] ? kthread+0x0/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.324234] [] ? child_rip+0x0/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.329979] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.351464] Call Trace: Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.354384] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.361858] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.369931] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.377598] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.385067] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.392664] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.400295] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.407663] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.414743] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.421673] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.429508] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.436689] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.445039] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.451444] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.458437] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.466374] [] ? strlcpy+0x4a/0x60 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.472011] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.478778] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.486129] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.493297] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.500648] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.508246] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.516286] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.523703] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.530732] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.538722] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.545867] [] ? thread_return+0x4e/0x7d0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.552216] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.559048] [] ? kthread+0x9e/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.564666] [] ? child_rip+0xa/0x20 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.570394] [] ? kthread+0x0/0xc0 Aug 25 20:47:27 feral28.ccs.ornl.gov kernel: [1485109.575896] [] ? child_rip+0x0/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.228202] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231202] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231203] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231245] CPU 5 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231246] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231281] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231283] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231286] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231290] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231292] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231293] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231295] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231297] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231298] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231300] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231303] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231304] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231306] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231308] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231310] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231311] Stack: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231312] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231314] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231316] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231318] Call Trace: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231362] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231409] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231458] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231505] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231552] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231567] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231578] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231590] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231601] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231616] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231632] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231661] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231676] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231691] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231741] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231745] [] ? strlcpy+0x4a/0x60 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231761] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231776] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231791] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231806] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231850] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231890] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231945] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.231999] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232052] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232104] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232108] [] ? thread_return+0x4e/0x7d0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232159] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232163] [] ? kthread+0x9e/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232166] [] ? child_rip+0xa/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232168] [] ? kthread+0x0/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232171] [] ? child_rip+0x0/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232172] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232187] Call Trace: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232230] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232276] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232326] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232372] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232419] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232434] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232445] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232457] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232467] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232483] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232499] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232527] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232542] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232557] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232607] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232611] [] ? strlcpy+0x4a/0x60 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232627] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232642] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232656] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232671] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232715] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232756] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232810] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232864] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232918] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232969] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.232973] [] ? thread_return+0x4e/0x7d0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.233025] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.233028] [] ? kthread+0x9e/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.233031] [] ? child_rip+0xa/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.233034] [] ? kthread+0x0/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.233036] [] ? child_rip+0x0/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235202] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235204] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235247] CPU 6 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235248] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235285] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235287] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235290] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235294] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235296] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235298] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235300] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235301] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235303] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235305] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235307] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235309] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235312] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235314] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235316] Stack: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235316] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235318] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235321] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235323] Call Trace: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235358] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235405] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235454] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235501] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235548] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235564] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235574] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235586] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235598] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235648] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235659] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235674] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235704] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235720] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235749] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235764] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235779] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235829] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235833] [] ? strlcpy+0x4a/0x60 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235850] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235865] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235879] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235894] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235938] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.235978] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236033] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236087] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236140] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236192] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236196] [] ? thread_return+0x4e/0x7d0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236247] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236251] [] ? kthread+0x9e/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236253] [] ? child_rip+0xa/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236256] [] ? kthread+0x0/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236258] [] ? child_rip+0x0/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236260] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236275] Call Trace: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236318] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236364] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236414] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236460] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236507] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236522] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236533] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236544] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236557] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236607] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236617] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236633] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236662] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236678] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236707] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236722] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236737] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236787] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236791] [] ? strlcpy+0x4a/0x60 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236807] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236822] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236837] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236852] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236895] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236936] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.236990] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.237044] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.237097] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.237149] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.237153] [] ? thread_return+0x4e/0x7d0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.237204] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.237208] [] ? kthread+0x9e/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.237211] [] ? child_rip+0xa/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.237213] [] ? kthread+0x0/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.237215] [] ? child_rip+0x0/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238201] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238202] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238243] CPU 7 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238243] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238280] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238282] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238285] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238289] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238290] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238292] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238293] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238295] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238297] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238299] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238300] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238302] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238304] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238305] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238307] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238309] Stack: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238310] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238312] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238314] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238317] Call Trace: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238351] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238398] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238447] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238494] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238541] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238556] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238567] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238579] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238589] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238604] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238621] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238650] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238665] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238680] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238730] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238734] [] ? strlcpy+0x4a/0x60 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238750] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238765] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238780] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238795] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238838] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238879] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238933] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.238987] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239041] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239092] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239096] [] ? thread_return+0x4e/0x7d0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239148] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239151] [] ? kthread+0x9e/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239154] [] ? child_rip+0xa/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239157] [] ? kthread+0x0/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239159] [] ? child_rip+0x0/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239160] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239176] Call Trace: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239218] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239265] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239314] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239361] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239408] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239423] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239434] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239446] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239456] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239471] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239487] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239516] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239531] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239546] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239596] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239600] [] ? strlcpy+0x4a/0x60 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239616] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239631] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239646] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239660] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239704] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239744] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239799] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239852] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239905] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239957] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.239961] [] ? thread_return+0x4e/0x7d0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.240012] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.240016] [] ? kthread+0x9e/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.240019] [] ? child_rip+0xa/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.240022] [] ? kthread+0x0/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485190.240024] [] ? child_rip+0x0/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485192.764707] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485192.862019] CPU 4 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485192.864125] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485192.961629] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485192.963502] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485192.977046] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485192.985662] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485192.991623] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485192.999512] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.007423] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.015312] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.023294] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.031251] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.040143] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.046531] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.054521] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.062469] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.070356] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.079820] Stack: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.082220] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.090141] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.098636] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.107477] Call Trace: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.110411] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.117809] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.125872] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.133479] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.140974] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.148537] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.156194] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.163639] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.170779] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.177734] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.185563] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.192770] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.200998] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.207385] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.214468] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.222403] [] ? strlcpy+0x4a/0x60 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.228091] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.234955] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.242305] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.249470] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.256738] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.264327] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.272357] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.279794] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.286719] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.294662] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.301775] [] ? thread_return+0x4e/0x7d0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.308090] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.314993] [] ? kthread+0x9e/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.320669] [] ? child_rip+0xa/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.326450] [] ? kthread+0x0/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.331969] [] ? child_rip+0x0/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.337741] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.359191] Call Trace: Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.362101] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.369514] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.377536] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.385177] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.392674] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.400190] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.407926] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.415319] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.422409] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.429347] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.437226] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.444459] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.452772] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.459162] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.466251] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.474179] [] ? strlcpy+0x4a/0x60 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.479859] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.486704] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.494080] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.501230] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.508529] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.516154] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.524229] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.531621] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.538615] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.546492] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.553585] [] ? thread_return+0x4e/0x7d0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.559944] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.566881] [] ? kthread+0x9e/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.572496] [] ? child_rip+0xa/0x20 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.578251] [] ? kthread+0x0/0xc0 Aug 25 20:48:51 feral28.ccs.ornl.gov kernel: [1485193.583743] [] ? child_rip+0x0/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.234805] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237806] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237807] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237849] CPU 5 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237850] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237885] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237887] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237890] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237894] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237896] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237897] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237899] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237900] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237902] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237904] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237906] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237907] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237909] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237911] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237913] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237914] Stack: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237915] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237917] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237919] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237922] Call Trace: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.237966] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238012] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238062] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238109] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238155] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238171] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238182] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238194] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238204] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238220] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238236] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238265] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238279] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238294] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238345] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238349] [] ? strlcpy+0x4a/0x60 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238365] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238380] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238395] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238410] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238454] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238494] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238548] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238602] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238656] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238707] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238711] [] ? thread_return+0x4e/0x7d0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238762] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238766] [] ? kthread+0x9e/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238768] [] ? child_rip+0xa/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238771] [] ? kthread+0x0/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238773] [] ? child_rip+0x0/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238775] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238790] Call Trace: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238833] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238879] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238929] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.238975] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239022] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239038] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239048] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239060] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239071] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239086] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239102] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239131] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239145] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239160] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239211] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239215] [] ? strlcpy+0x4a/0x60 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239231] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239245] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239260] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239275] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239319] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239359] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239413] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239467] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239521] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239572] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239576] [] ? thread_return+0x4e/0x7d0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239628] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239631] [] ? kthread+0x9e/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239634] [] ? child_rip+0xa/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239636] [] ? kthread+0x0/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.239639] [] ? child_rip+0x0/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241806] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241808] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241850] CPU 6 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241850] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241886] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241888] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241891] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241896] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241898] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241899] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241901] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241903] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241904] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241906] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241908] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241910] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241913] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241915] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241917] Stack: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241918] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241920] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241922] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241924] Call Trace: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.241959] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242005] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242055] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242102] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242149] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242165] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242175] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242187] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242199] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242250] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242260] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242276] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242307] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242323] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242352] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242367] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242382] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242432] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242436] [] ? strlcpy+0x4a/0x60 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242452] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242467] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242482] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242497] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242540] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242581] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242635] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242689] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242744] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242796] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242799] [] ? thread_return+0x4e/0x7d0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242852] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242855] [] ? kthread+0x9e/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242858] [] ? child_rip+0xa/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242861] [] ? kthread+0x0/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242863] [] ? child_rip+0x0/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242864] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242880] Call Trace: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242923] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.242969] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243019] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243065] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243112] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243127] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243138] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243150] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243162] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243212] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243223] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243238] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243269] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243286] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243314] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243329] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243344] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243394] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243398] [] ? strlcpy+0x4a/0x60 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243414] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243429] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243443] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243458] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243502] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243542] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243597] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243651] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243704] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243756] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243759] [] ? thread_return+0x4e/0x7d0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243811] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243815] [] ? kthread+0x9e/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243817] [] ? child_rip+0xa/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243820] [] ? kthread+0x0/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.243822] [] ? child_rip+0x0/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244804] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244806] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244845] CPU 7 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244846] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244883] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244884] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244887] RIP: 0010:[] [] _spin_lock+0x18/0x30 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244891] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244893] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244894] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244896] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244897] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244899] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244901] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244903] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244904] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244906] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244908] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244910] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244911] Stack: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244912] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244914] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244916] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244919] Call Trace: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244953] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.244999] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245049] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245096] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245142] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245158] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245169] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245181] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245191] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245206] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245223] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245252] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245266] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245281] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245332] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245336] [] ? strlcpy+0x4a/0x60 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245352] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245367] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245382] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245396] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245440] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245480] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245534] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245588] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245642] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245693] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245697] [] ? thread_return+0x4e/0x7d0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245749] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245752] [] ? kthread+0x9e/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245755] [] ? child_rip+0xa/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245758] [] ? kthread+0x0/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245760] [] ? child_rip+0x0/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245761] Code: 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 0e f3 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245776] Call Trace: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245819] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245865] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245915] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.245961] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246008] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246024] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246035] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246046] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246057] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246072] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246088] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246117] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246132] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246146] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246197] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246201] [] ? strlcpy+0x4a/0x60 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246217] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246232] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246246] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246261] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246305] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246345] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246399] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246453] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246507] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246558] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246562] [] ? thread_return+0x4e/0x7d0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246613] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246617] [] ? kthread+0x9e/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246620] [] ? child_rip+0xa/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246623] [] ? kthread+0x0/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485274.246625] [] ? child_rip+0x0/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485276.769433] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485276.866513] CPU 4 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485276.868619] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485276.965904] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485276.967769] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485276.981114] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485276.989835] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485276.995772] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.003763] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.011666] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.019606] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.027520] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.035424] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.044366] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.050684] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.058625] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.066555] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.074521] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.083993] Stack: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.086384] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.094332] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.102913] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.111727] Call Trace: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.114663] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.122025] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.130106] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.137791] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.145288] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.152841] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.160515] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.167831] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.174910] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.181823] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.189753] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.196882] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.205257] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.211629] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.218676] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.226655] [] ? strlcpy+0x4a/0x60 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.232273] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.239102] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.246461] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.253611] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.260937] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.268508] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.276581] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.284027] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.290961] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.298837] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.305879] [] ? thread_return+0x4e/0x7d0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.312202] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.319165] [] ? kthread+0x9e/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.324798] [] ? child_rip+0xa/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.330562] [] ? kthread+0x0/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.336064] [] ? child_rip+0x0/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.341792] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.363251] Call Trace: Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.366153] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.373541] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.381570] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.389271] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.396743] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.404269] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.411848] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.419243] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.426339] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.433226] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.441104] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.448328] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.456627] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.462996] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.469990] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.477927] [] ? strlcpy+0x4a/0x60 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.483538] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.490349] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.497740] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.504962] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.512289] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.519892] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.530681] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.538134] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.545093] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.552979] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.560046] [] ? thread_return+0x4e/0x7d0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.566361] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.573281] [] ? kthread+0x9e/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.578905] [] ? child_rip+0xa/0x20 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.584633] [] ? kthread+0x0/0xc0 Aug 25 20:50:15 feral28.ccs.ornl.gov kernel: [1485277.590154] [] ? child_rip+0x0/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.241409] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244409] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244410] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244452] CPU 5 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244453] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244489] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244490] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244493] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244497] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244499] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244501] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244502] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244504] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244505] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244507] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244509] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244511] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244514] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244516] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244517] Stack: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244518] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244520] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244523] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244525] Call Trace: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244569] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244616] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244666] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244712] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244759] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244775] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244786] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244798] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244808] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244823] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244840] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244869] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244883] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244898] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244949] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244953] [] ? strlcpy+0x4a/0x60 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244969] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244984] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.244999] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245014] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245058] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245098] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245152] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245206] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245260] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245312] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245316] [] ? thread_return+0x4e/0x7d0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245368] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245371] [] ? kthread+0x9e/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245374] [] ? child_rip+0xa/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245377] [] ? kthread+0x0/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245379] [] ? child_rip+0x0/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245380] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245395] Call Trace: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245437] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245484] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245533] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245580] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245627] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245642] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245653] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245665] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245675] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245690] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245707] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245735] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245750] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245765] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245815] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245819] [] ? strlcpy+0x4a/0x60 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245835] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245850] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245865] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245879] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245923] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.245963] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246018] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246072] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246125] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246177] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246181] [] ? thread_return+0x4e/0x7d0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246232] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246236] [] ? kthread+0x9e/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246239] [] ? child_rip+0xa/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246241] [] ? kthread+0x0/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.246243] [] ? child_rip+0x0/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248409] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248410] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248452] CPU 6 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248453] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248489] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248491] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248493] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248498] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248499] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248501] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248503] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248504] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248506] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248508] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248510] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248512] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248514] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248515] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248518] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248519] Stack: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248520] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248522] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248524] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248526] Call Trace: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248561] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248608] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248657] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248704] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248751] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248767] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248777] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248789] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248801] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248852] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248862] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248878] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248909] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248926] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248954] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248969] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.248984] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249034] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249038] [] ? strlcpy+0x4a/0x60 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249054] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249069] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249084] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249099] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249143] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249183] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249237] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249291] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249345] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249397] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249400] [] ? thread_return+0x4e/0x7d0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249452] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249456] [] ? kthread+0x9e/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249458] [] ? child_rip+0xa/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249461] [] ? kthread+0x0/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249463] [] ? child_rip+0x0/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249465] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249480] Call Trace: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249523] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249569] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249619] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249665] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249712] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249727] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249738] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249749] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249762] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249812] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249823] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249838] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249869] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249885] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249914] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249928] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249943] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249994] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.249998] [] ? strlcpy+0x4a/0x60 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250014] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250028] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250043] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250058] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250102] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250142] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250196] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250250] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250304] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250356] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250360] [] ? thread_return+0x4e/0x7d0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250411] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250415] [] ? kthread+0x9e/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250418] [] ? child_rip+0xa/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250420] [] ? kthread+0x0/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.250423] [] ? child_rip+0x0/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251408] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251410] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251449] CPU 7 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251450] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251487] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251488] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251491] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251495] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251497] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251499] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251500] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251502] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251503] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251505] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251507] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251509] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251513] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251515] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251516] Stack: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251517] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251519] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251521] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251524] Call Trace: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251557] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251604] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251653] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251700] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251747] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251762] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251773] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251785] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251796] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251811] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251827] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251856] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251870] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251885] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251935] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251939] [] ? strlcpy+0x4a/0x60 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251955] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251970] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.251985] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252000] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252044] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252084] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252138] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252192] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252246] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252298] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252301] [] ? thread_return+0x4e/0x7d0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252353] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252356] [] ? kthread+0x9e/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252359] [] ? child_rip+0xa/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252362] [] ? kthread+0x0/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252364] [] ? child_rip+0x0/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252365] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252380] Call Trace: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252423] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252469] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252519] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252565] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252612] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252627] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252638] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252650] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252660] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252675] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252691] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252720] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252735] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252750] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252800] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252804] [] ? strlcpy+0x4a/0x60 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252820] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252835] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252849] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252864] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252908] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.252948] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253002] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253056] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253110] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253161] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253165] [] ? thread_return+0x4e/0x7d0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253216] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253220] [] ? kthread+0x9e/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253223] [] ? child_rip+0xa/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253226] [] ? kthread+0x0/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485358.253228] [] ? child_rip+0x0/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485360.775956] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485360.873493] CPU 4 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485360.875591] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485360.973000] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485360.974872] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485360.988297] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485360.996988] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.002951] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.010892] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.018814] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.026718] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.034631] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.042552] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.051436] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.057746] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.065719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.073650] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.081639] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.091112] Stack: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.093460] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.101322] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.109807] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.118674] Call Trace: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.121585] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.129006] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.137061] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.144711] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.152208] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.159735] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.167314] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.174691] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.181787] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.188674] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.196561] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.203716] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.212056] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.218451] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.225491] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.233419] [] ? strlcpy+0x4a/0x60 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.239082] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.245970] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.253372] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.260548] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.267901] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.275525] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.283564] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.291061] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.300671] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.308540] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.315650] [] ? thread_return+0x4e/0x7d0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.321931] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.328928] [] ? kthread+0x9e/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.334648] [] ? child_rip+0xa/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.340386] [] ? kthread+0x0/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.345914] [] ? child_rip+0x0/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.351588] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.373049] Call Trace: Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.376012] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.383434] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.391688] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.399398] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.406922] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.414509] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.422183] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.429587] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.436758] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.443933] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.451838] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.459036] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.467351] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.473705] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.480768] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.488748] [] ? strlcpy+0x4a/0x60 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.494351] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.501152] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.508553] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.515712] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.523038] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.530594] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.538606] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.545921] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.552916] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.560810] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.567894] [] ? thread_return+0x4e/0x7d0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.574209] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.581103] [] ? kthread+0x9e/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.586736] [] ? child_rip+0xa/0x20 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.592430] [] ? kthread+0x0/0xc0 Aug 25 20:51:39 feral28.ccs.ornl.gov kernel: [1485361.597905] [] ? child_rip+0x0/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.248013] BUG: soft lockup - CPU#4 stuck for 67s! [mdt01_002:21487] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251013] BUG: soft lockup - CPU#5 stuck for 67s! [mdt01_007:17158] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251014] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251056] CPU 5 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251057] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251092] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251094] Pid: 17158, comm: mdt01_007 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251097] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251101] RSP: 0018:ffff880bbf14f6d0 EFLAGS: 00000297 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251103] RAX: 000000000000000c RBX: ffff880bbf14f6d0 RCX: ffff8817344e8608 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251104] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251106] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bbf14f890 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251107] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bbf14f650 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251109] R13: ffff88176c6f64c0 R14: ffffffffa0ba7520 R15: ffff880beaae46b8 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251111] FS: 0000000000000000(0000) GS:ffff880c6a620000(0000) knlGS:0000000000000000 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251113] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251114] CR2: 00007f91f4e41000 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251118] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251120] Process mdt01_007 (pid: 17158, threadinfo ffff880bbf14c000, task ffff880beaae4040) Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251121] Stack: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251122] ffff880bbf14f6f0 ffffffffa0a62070 ffff880bbf14f6f0 ffff88182b2d8c00 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251124] ffff880bbf14f790 ffffffffa0a83646 ffff881734b2cd70 ffff881734b2cd70 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251126] 00000000ebc0de04 ffff881734b2cd48 ffff880bbf14f770 ffffffffa0aa42eb Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251129] Call Trace: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251173] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251220] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251269] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251316] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251363] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251379] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251390] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251402] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251412] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251428] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251444] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251473] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251488] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251503] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251553] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251557] [] ? strlcpy+0x4a/0x60 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251573] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251588] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251603] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251618] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251662] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251702] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251757] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251811] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251865] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251917] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251920] [] ? thread_return+0x4e/0x7d0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251972] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251975] [] ? kthread+0x9e/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251978] [] ? child_rip+0xa/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251981] [] ? kthread+0x0/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251983] [] ? child_rip+0x0/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251984] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.251999] Call Trace: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252042] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252088] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252138] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252184] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252231] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252246] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252257] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252269] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252279] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252294] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252310] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252339] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252354] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252369] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252419] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252423] [] ? strlcpy+0x4a/0x60 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252439] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252454] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252469] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252484] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252528] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252568] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252622] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252676] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252730] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252782] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252785] [] ? thread_return+0x4e/0x7d0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252837] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252840] [] ? kthread+0x9e/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252843] [] ? child_rip+0xa/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252846] [] ? kthread+0x0/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.252848] [] ? child_rip+0x0/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255013] BUG: soft lockup - CPU#6 stuck for 67s! [mdt01_000:21485] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255014] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255057] CPU 6 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255058] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255094] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255095] Pid: 21485, comm: mdt01_000 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255098] RIP: 0010:[] [] _spin_lock+0x21/0x30 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255103] RSP: 0018:ffff880bc79f36d0 EFLAGS: 00000297 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255105] RAX: 0000000000000009 RBX: ffff880bc79f36d0 RCX: ffff8817c7e23f48 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255106] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255108] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79f3890 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255109] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79f3650 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255111] R13: ffff8817349ce4c0 R14: ffffffffa0ba7520 R15: ffff880bc79e26b8 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255113] FS: 0000000000000000(0000) GS:ffff880c6a640000(0000) knlGS:0000000000000000 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255115] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255117] CR2: 00007f91f4b730c0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255123] Process mdt01_000 (pid: 21485, threadinfo ffff880bc79f0000, task ffff880bc79e2040) Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255124] Stack: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255125] ffff880bc79f36f0 ffffffffa0a62070 ffff880bc79f36f0 ffff88182b2d8c00 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255127] ffff880bc79f3790 ffffffffa0a83646 ffff881734a84d70 ffff881734a84d70 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255129] 00000000ebc0de04 ffff881734a84d48 ffff880bc79f3770 ffffffffa0aa42eb Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255132] Call Trace: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255166] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255213] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255263] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255309] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255356] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255372] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255382] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255394] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255406] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255457] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255467] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255483] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255514] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255531] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255559] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255574] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255589] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255640] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255644] [] ? strlcpy+0x4a/0x60 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255660] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255675] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255689] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255704] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255748] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255788] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255842] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255896] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.255950] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256002] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256005] [] ? thread_return+0x4e/0x7d0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256057] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256060] [] ? kthread+0x9e/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256063] [] ? child_rip+0xa/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256066] [] ? kthread+0x0/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256068] [] ? child_rip+0x0/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256069] Code: 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256084] Call Trace: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256127] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256174] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256223] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256270] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256316] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256332] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256343] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256354] [] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256366] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256416] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256427] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256442] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256471] [] ? lu_object_find+0x16/0x20 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256487] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256516] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256530] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256545] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256596] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256599] [] ? strlcpy+0x4a/0x60 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256616] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256630] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256645] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256660] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256704] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256744] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256798] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256852] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256906] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256958] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.256961] [] ? thread_return+0x4e/0x7d0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.257013] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.257016] [] ? kthread+0x9e/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.257019] [] ? child_rip+0xa/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.257022] [] ? kthread+0x0/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.257024] [] ? child_rip+0x0/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258012] BUG: soft lockup - CPU#7 stuck for 67s! [mdt01_001:21486] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258013] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258054] CPU 7 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258055] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258091] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258093] Pid: 21486, comm: mdt01_001 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258096] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258100] RSP: 0018:ffff880bc79fb6d0 EFLAGS: 00000293 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258101] RAX: 000000000000000b RBX: ffff880bc79fb6d0 RCX: ffff88176c6dd348 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258103] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258105] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79fb890 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258106] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79fb650 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258108] R13: ffff8817349ef4c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5b98 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258110] FS: 0000000000000000(0000) GS:ffff880c6a660000(0000) knlGS:0000000000000000 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258112] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258113] CR2: 00007fcbdae23380 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258115] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258116] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258118] Process mdt01_001 (pid: 21486, threadinfo ffff880bc79f8000, task ffff880bc79f5520) Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258119] Stack: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258120] ffff880bc79fb6f0 ffffffffa0a62070 ffff880bc79fb6f0 ffff88182b2d8c00 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258122] ffff880bc79fb790 ffffffffa0a83646 ffff881734b996b0 ffff881734b996b0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258124] 00000000ebc0de04 ffff881734b99688 ffff880bc79fb770 ffffffffa0aa42eb Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258127] Call Trace: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258161] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258207] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258257] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258303] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258350] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258366] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258376] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258388] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258399] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258414] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258430] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258459] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258474] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258489] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258539] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258543] [] ? strlcpy+0x4a/0x60 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258559] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258574] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258588] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258603] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258647] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258688] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258741] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258796] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258849] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258901] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258905] [] ? thread_return+0x4e/0x7d0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258956] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258960] [] ? kthread+0x9e/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258963] [] ? child_rip+0xa/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258966] [] ? kthread+0x0/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258968] [] ? child_rip+0x0/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258969] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.258984] Call Trace: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259026] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259073] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259122] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259169] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259215] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259231] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259241] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259253] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259264] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259279] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259295] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259324] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259338] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259353] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259403] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259407] [] ? strlcpy+0x4a/0x60 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259423] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259438] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259453] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259468] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259511] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259552] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259606] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259660] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259713] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259765] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259768] [] ? thread_return+0x4e/0x7d0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259820] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259824] [] ? kthread+0x9e/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259826] [] ? child_rip+0xa/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259829] [] ? kthread+0x0/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485442.259831] [] ? child_rip+0x0/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485444.780432] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485444.877650] CPU 4 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485444.879763] Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) mbcache jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ptlrpc(U) obdclass(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) autofs4 ipmi_devintf ipmi_si ipmi_msghandler 8021q garp stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_sa ib_mad ib_core ib_addr zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) raid0 dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg microcode sd_mod crc_t10dif iTCO_wdt iTCO_vendor_support joydev mpt2sas scsi_transport_sas raid_class serio_raw i2c_i801 i2c_core ahci lpc_ich mfd_core ioatdma dca i7core_edac edac_core shpchp ipv6 nfs lockd fscache auth_rpcgss nfs_acl sunrpc e1000e mlx4_en ptp pps_core mlx4_core [la Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: st unloaded: scsi_wait_scan] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485444.977311] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485444.979175] Pid: 21487, comm: mdt01_002 Tainted: P --L------------ 2.6.32-573.26.1.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485444.992661] RIP: 0010:[] [] _spin_lock+0x1e/0x30 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.001328] RSP: 0018:ffff880bc79ff6d0 EFLAGS: 00000293 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.007229] RAX: 000000000000000a RBX: ffff880bc79ff6d0 RCX: ffff88173407ff48 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.015159] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff8817349cb79c Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.023039] RBP: ffffffff8100bc0e R08: 0000000000000003 R09: ffff880bc79ff890 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.030977] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880bc79ff650 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.038976] R13: ffff881734a410c0 R14: ffffffffa0ba7520 R15: ffff880bc79f5128 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.046882] FS: 0000000000000000(0000) GS:ffff880c6a600000(0000) knlGS:0000000000000000 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.055834] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.062239] CR2: 0000000000ef46f0 CR3: 0000000001a8d000 CR4: 00000000000007e0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.070185] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.078126] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.088691] Process mdt01_002 (pid: 21487, threadinfo ffff880bc79fc000, task ffff880bc79f4ab0) Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.098189] Stack: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.100616] ffff880bc79ff6f0 ffffffffa0a62070 ffff880bc79ff6f0 ffff88182b2d8c00 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.108538] ffff880bc79ff790 ffffffffa0a83646 ffff881734aa1f30 ffff881734aa1f30 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.116971] 00000000ebc0de04 ffff881734aa1f08 ffff880bc79ff770 ffffffffa0aa42eb Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.125810] Call Trace: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.128757] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.136170] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.144259] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.151970] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.159449] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.167019] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.174945] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.182252] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.189348] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.196236] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.204045] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.211198] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.219402] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.225727] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.232655] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.240503] [] ? strlcpy+0x4a/0x60 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.246158] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.253029] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.260344] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.267556] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.274891] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.282462] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.290553] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.297980] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.304906] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.312773] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.319839] [] ? thread_return+0x4e/0x7d0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.326208] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.333118] [] ? kthread+0x9e/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.338664] [] ? child_rip+0xa/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.344376] [] ? kthread+0x0/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.349923] [] ? child_rip+0x0/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.355573] Code: 00 00 00 01 74 05 e8 32 19 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.377015] Call Trace: Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.379959] [] ? lock_res_and_lock+0x30/0x50 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.387365] [] ? ldlm_cli_enqueue_fini+0x206/0xe30 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.395454] [] ? ptlrpc_set_destroy+0x26b/0x450 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.403112] [] ? ldlm_cli_enqueue+0x3c1/0x870 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.410652] [] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.418205] [] ? mdt_remote_blocking_ast+0x0/0x4c0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.425818] [] ? osp_md_object_lock+0x16b/0x1f0 [osp] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.433195] [] ? lod_object_lock+0x14f/0x900 [lod] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.440327] [] ? mdd_object_lock+0x3b/0xd0 [mdd] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.447214] [] ? mdt_remote_object_lock+0x15a/0x4a0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.455041] [] ? mdt_reint_open+0x2b90/0x3180 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.462334] [] ? upcall_cache_get_entry+0x29c/0x880 [obdclass] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.470657] [] ? mdt_ucred+0x15/0x20 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.477017] [] ? mdt_root_squash+0x2c/0x3f0 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.484049] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.491985] [] ? strlcpy+0x4a/0x60 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.497603] [] ? mdt_reint_rec+0x5d/0x200 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.504476] [] ? mdt_reint_internal+0x62b/0xa50 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.511869] [] ? mdt_intent_reint+0x1f6/0x440 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.519097] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.526370] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.533951] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.542008] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.549410] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.556326] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.564185] [] ? ptlrpc_main+0xd31/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.571355] [] ? thread_return+0x4e/0x7d0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.577731] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.584608] [] ? kthread+0x9e/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.590233] [] ? child_rip+0xa/0x20 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.595918] [] ? kthread+0x0/0xc0 Aug 25 20:53:03 feral28.ccs.ornl.gov kernel: [1485445.601421] [] ? child_rip+0x0/0x20 Aug 25 20:53:17 feral28.ccs.ornl.gov kernel: [1485460.256358] Lustre: sultan-MDT000b: Client 2d302f5b-b294-591e-d666-ff70376d37f3 (at 10@gni1) reconnecting Aug 25 20:53:17 feral28.ccs.ornl.gov kernel: [1485460.266757] Lustre: Skipped 8 previous similar messages Aug 25 20:53:17 feral28.ccs.ornl.gov kernel: [1485460.272553] Lustre: sultan-MDT000b: Connection restored to 61d60a25-2cca-787d-3160-3ac0a705962b (at 10@gni1) Aug 25 20:53:17 feral28.ccs.ornl.gov kernel: [1485460.283305] Lustre: Skipped 10 previous similar messages