[11694.671469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == parallel-scale test mdtestfpp: mdtestfpp ========================================================== 12:43:53 \(1434051833\) [11695.035400] Lustre: DEBUG MARKER: == parallel-scale test mdtestfpp: mdtestfpp ========================================================== 12:43:53 (1434051833) [12064.045003] BUG: soft lockup - CPU#1 stuck for 22s! [mdt00_006:31733] [12064.045003] Modules linked in: osp(OF) mdd(OF) lod(OF) mdt(OF) lfsck(OF) mgs(OF) mgc(OF) osd_ldiskfs(OF) lquota(OF) fid(OF) fld(OF) ksocklnd(OF) ptlrpc(OF) obdclass(OF) lnet(OF) sha512_generic libcfs(OF) ldiskfs(OF) dm_mod nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd fscache xprtrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ppdev ib_sa pcspkr virtio_balloon serio_raw parport_pc parport i2c_piix4 ib_mad ib_core ib_addr ext4 mbcache jbd2 ata_generic pata_acpi cirrus virtio_blk syscopyarea sysfillrect sysimgblt drm_kms_helper ttm ata_piix drm 8139too virtio_pci 8139cp virtio_ring virtio mii i2c_core libata floppy [12064.045003] CPU: 1 PID: 31733 Comm: mdt00_006 Tainted: GF O-------------- 3.10.0-229.4.2.el7_lustre.g1fee634.x86_64 #1 [12064.045003] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [12064.045003] task: ffff880079498000 ti: ffff88004bd64000 task.ti: ffff88004bd64000 [12064.045003] RIP: 0010:[<ffffffffa0bc8b9c>] [<ffffffffa0bc8b9c>] iam_insert_key_lock+0x2c/0x50 [osd_ldiskfs] [12064.045003] RSP: 0018:ffff88004bd673d0 EFLAGS: 00000206 [12064.045003] RAX: 000000000231c029 RBX: ffff88007b1e7240 RCX: 00000000000000cc [12064.045003] RDX: ffff8800120137fc RSI: ffff88004bd67500 RDI: ffff88004bd674c8 [12064.045003] RBP: ffff88004bd673d8 R08: ffff88005bfa23a8 R09: 0000000000000055 [12064.045003] R10: ffff880079ecd1c8 R11: 7d1b000002000000 R12: ffff88004bd67360 [12064.045003] R13: 0000000000000000 R14: ffff88005662b930 R15: ffffffffa05c4e87 [12064.045003] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 [12064.045003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [12064.045003] CR2: 00007f3dc08ef000 CR3: 000000007874c000 CR4: 00000000000006e0 [12064.045003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12064.045003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12064.045003] Stack: [12064.045003] ffff88004bd675a8 ffff88004bd67430 ffffffffa0bcb582 ffff88004bd67478 [12064.045003] ffff8800583c19c0 000000aa4bd6746c ffff880015727000 ffff88004bd674c8 [12064.045003] ffff88007b1e7240 ffff88005d72a338 ffff88004bd675a8 ffff88005d72a360 [12064.045003] Call Trace: [12064.045003] [<ffffffffa0bcb582>] iam_lfix_split+0x122/0x140 [osd_ldiskfs] [12064.045003] [<ffffffffa0bc9e16>] iam_add_rec+0x236/0x2f0 [osd_ldiskfs] [12064.045003] [<ffffffffa0bcb5a0>] ? iam_lfix_split+0x140/0x140 [osd_ldiskfs] [12064.045003] [<ffffffffa0bca77e>] iam_insert+0xce/0x120 [osd_ldiskfs] [12064.045003] [<ffffffffa0bc2115>] osd_oi_iam_refresh.isra.16+0x125/0x2a0 [osd_ldiskfs] [12064.045003] [<ffffffffa0bc44c7>] osd_oi_insert+0x147/0x490 [osd_ldiskfs] [12064.045003] [<ffffffff811eb4b2>] ? generic_setxattr+0x62/0x80 [12064.045003] [<ffffffffa0bbf767>] osd_object_ea_create+0x527/0x980 [osd_ldiskfs] [12064.045003] [<ffffffffa0e4cebc>] lod_sub_object_create+0x22c/0x460 [lod] [12064.045003] [<ffffffffa0e4495f>] lod_object_create+0xaf/0x200 [lod] [12064.045003] [<ffffffffa0ea62e5>] mdd_object_create_internal+0xb5/0x280 [mdd] [12064.045003] [<ffffffffa0e91b96>] mdd_object_create+0x76/0xa20 [mdd] [12064.045003] [<ffffffffa0e9ac17>] ? mdd_declare_create+0x447/0xd30 [mdd] [12064.045003] [<ffffffffa0e9c1c0>] mdd_create+0xcc0/0x1260 [mdd] [12064.045003] [<ffffffffa0d8101b>] mdt_reint_open+0x1f4b/0x2da0 [mdt] [12064.045003] [<ffffffffa0784149>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] [12064.045003] [<ffffffff812ddd72>] ? strlcpy+0x42/0x60 [12064.045003] [<ffffffffa0d75920>] mdt_reint_rec+0x80/0x210 [mdt] [12064.045003] [<ffffffffa0d591ac>] mdt_reint_internal+0x58c/0x780 [mdt] [12064.045003] [<ffffffffa0d59502>] mdt_intent_reint+0x162/0x420 [mdt] [12064.045003] [<ffffffffa0d630ba>] mdt_intent_policy+0x58a/0xb80 [mdt] [12064.045003] [<ffffffffa0972103>] ldlm_lock_enqueue+0x353/0x930 [ptlrpc] [12064.045003] [<ffffffffa0999062>] ldlm_handle_enqueue0+0x4f2/0x1540 [ptlrpc] [12064.045003] [<ffffffffa09beeb0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [12064.045003] [<ffffffffa0a14562>] tgt_enqueue+0x62/0x210 [ptlrpc] [12064.045003] [<ffffffffa0a18235>] tgt_request_handle+0x6d5/0x1060 [ptlrpc] [12064.045003] [<ffffffffa09c820b>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] [12064.045003] [<ffffffffa09c5d88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [12064.045003] [<ffffffff810a9662>] ? default_wake_function+0x12/0x20 [12064.045003] [<ffffffff810a0898>] ? __wake_up_common+0x58/0x90 [12064.045003] [<ffffffffa09cc3f8>] ptlrpc_main+0xaf8/0x1ea0 [ptlrpc] [12064.045003] [<ffffffff810ad8b6>] ? __dequeue_entity+0x26/0x40 [12064.045003] [<ffffffff810125f6>] ? __switch_to+0x136/0x4a0 [12064.045003] [<ffffffffa09cb900>] ? ptlrpc_register_service+0xf00/0xf00 [ptlrpc] [12064.045003] [<ffffffff8109739f>] kthread+0xcf/0xe0 [12064.045003] [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 [12064.045003] [<ffffffff81614f7c>] ret_from_fork+0x7c/0xb0 [12064.045003] [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 [12064.045003] Code: 1f 44 00 00 55 48 89 e5 53 4c 8b 06 48 89 f3 f0 41 0f ba 28 19 19 c0 85 c0 74 12 0f 1f 40 00 49 8b 00 a9 00 00 00 02 74 e6 f3 90 <eb> f2 48 89 de e8 2a ff ff ff 48 8b 1b e8 22 7a 4d e0 f0 80 63 [12064.045003] Kernel panic - not syncing: softlockup: hung tasks [12064.045003] CPU: 1 PID: 31733 Comm: mdt00_006 Tainted: GF O-------------- 3.10.0-229.4.2.el7_lustre.g1fee634.x86_64 #1 [12064.045003] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [12064.045003] ffffffff818371a7 000000005c8a5440 ffff88007fd03e18 ffffffff816050da [12064.045003] ffff88007fd03e98 ffffffff815fe94e 0000000000000008 ffff88007fd03ea8 [12064.045003] ffff88007fd03e48 000000005c8a5440 ffff88007fd03e67 0000000000000046 [12064.045003] Call Trace: [12064.045003] <IRQ> [<ffffffff816050da>] dump_stack+0x19/0x1b [12064.045003] [<ffffffff815fe94e>] panic+0xd8/0x1e7 [12064.045003] [<ffffffff8110a805>] watchdog_timer_fn+0x165/0x170 [12064.045003] [<ffffffff8109b2b7>] __run_hrtimer+0x77/0x1d0 [12064.045003] [<ffffffff8110a6a0>] ? watchdog_cleanup+0x10/0x10 [12064.045003] [<ffffffff8109baf7>] hrtimer_interrupt+0xf7/0x240 [12064.045003] [<ffffffff81044317>] local_apic_timer_interrupt+0x37/0x60 [12064.045003] [<ffffffff816175cf>] smp_apic_timer_interrupt+0x3f/0x60 [12064.045003] [<ffffffff81615c9d>] apic_timer_interrupt+0x6d/0x80 [12064.045003] <EOI> [<ffffffffa0bc8b9c>] ? iam_insert_key_lock+0x2c/0x50 [osd_ldiskfs] [12064.045003] [<ffffffffa0bcb582>] iam_lfix_split+0x122/0x140 [osd_ldiskfs] [12064.045003] [<ffffffffa0bc9e16>] iam_add_rec+0x236/0x2f0 [osd_ldiskfs] [12064.045003] [<ffffffffa0bcb5a0>] ? iam_lfix_split+0x140/0x140 [osd_ldiskfs] [12064.045003] [<ffffffffa0bca77e>] iam_insert+0xce/0x120 [osd_ldiskfs] [12064.045003] [<ffffffffa0bc2115>] osd_oi_iam_refresh.isra.16+0x125/0x2a0 [osd_ldiskfs] [12064.045003] [<ffffffffa0bc44c7>] osd_oi_insert+0x147/0x490 [osd_ldiskfs] [12064.045003] [<ffffffff811eb4b2>] ? generic_setxattr+0x62/0x80 [12064.045003] [<ffffffffa0bbf767>] osd_object_ea_create+0x527/0x980 [osd_ldiskfs] [12064.045003] [<ffffffffa0e4cebc>] lod_sub_object_create+0x22c/0x460 [lod] [12064.045003] [<ffffffffa0e4495f>] lod_object_create+0xaf/0x200 [lod] [12064.045003] [<ffffffffa0ea62e5>] mdd_object_create_internal+0xb5/0x280 [mdd] [12064.045003] [<ffffffffa0e91b96>] mdd_object_create+0x76/0xa20 [mdd] [12064.045003] [<ffffffffa0e9ac17>] ? mdd_declare_create+0x447/0xd30 [mdd] [12064.045003] [<ffffffffa0e9c1c0>] mdd_create+0xcc0/0x1260 [mdd] [12064.045003] [<ffffffffa0d8101b>] mdt_reint_open+0x1f4b/0x2da0 [mdt] [12064.045003] [<ffffffffa0784149>] ? upcall_cache_get_entry+0x3e9/0x8e0 [obdclass] [12064.045003] [<ffffffff812ddd72>] ? strlcpy+0x42/0x60 [12064.045003] [<ffffffffa0d75920>] mdt_reint_rec+0x80/0x210 [mdt] [12064.045003] [<ffffffffa0d591ac>] mdt_reint_internal+0x58c/0x780 [mdt] [12064.045003] [<ffffffffa0d59502>] mdt_intent_reint+0x162/0x420 [mdt] [12064.045003] [<ffffffffa0d630ba>] mdt_intent_policy+0x58a/0xb80 [mdt] [12064.045003] [<ffffffffa0972103>] ldlm_lock_enqueue+0x353/0x930 [ptlrpc] [12064.045003] [<ffffffffa0999062>] ldlm_handle_enqueue0+0x4f2/0x1540 [ptlrpc] [12064.045003] [<ffffffffa09beeb0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [12064.045003] [<ffffffffa0a14562>] tgt_enqueue+0x62/0x210 [ptlrpc] [12064.045003] [<ffffffffa0a18235>] tgt_request_handle+0x6d5/0x1060 [ptlrpc] [12064.045003] [<ffffffffa09c820b>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] [12064.045003] [<ffffffffa09c5d88>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [12064.045003] [<ffffffff810a9662>] ? default_wake_function+0x12/0x20 [12064.045003] [<ffffffff810a0898>] ? __wake_up_common+0x58/0x90 [12064.045003] [<ffffffffa09cc3f8>] ptlrpc_main+0xaf8/0x1ea0 [ptlrpc] [12064.045003] [<ffffffff810ad8b6>] ? __dequeue_entity+0x26/0x40 [12064.045003] [<ffffffff810125f6>] ? __switch_to+0x136/0x4a0 [12064.045003] [<ffffffffa09cb900>] ? ptlrpc_register_service+0xf00/0xf00 [ptlrpc] [12064.045003] [<ffffffff8109739f>] kthread+0xcf/0xe0 [12064.045003] [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 [12064.045003] [<ffffffff81614f7c>] ret_from_fork+0x7c/0xb0 [12064.045003] [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 [12064.045003] drm_kms_helper: panic occurred, switching back to text console [12064.101249] ------------[ cut here ]------------ [12064.101249] WARNING: at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5f/0x70() [12064.101249] Modules linked in: osp(OF) mdd(OF) lod(OF) mdt(OF) lfsck(OF) mgs(OF) mgc(OF) osd_ldiskfs(OF) lquota(OF) fid(OF) fld(OF) ksocklnd(OF) ptlrpc(OF) obdclass(OF) lnet(OF) sha512_generic libcfs(OF) ldiskfs(OF) dm_mod nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd fscache xprtrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ppdev ib_sa pcspkr virtio_balloon serio_raw parport_pc parport i2c_piix4 ib_mad ib_core ib_addr ext4 mbcache jbd2 ata_generic pata_acpi cirrus virtio_blk syscopyarea sysfillrect sysimgblt drm_kms_helper ttm ata_piix drm 8139too virtio_pci 8139cp virtio_ring virtio mii i2c_core libata floppy [12064.101249] CPU: 1 PID: 16 Comm: migration/1 Tainted: GF O-------------- 3.10.0-229.4.2.el7_lustre.g1fee634.x86_64 #1 [12064.101249] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [12064.101249] 0000000000000000 000000007b42a504 ffff88007c073c68 ffffffff816050da [12064.101249] ffff88007c073ca0 ffffffff8106e34b ffff88007fc13680 ffff88007c073d20 [12064.101249] ffff8800794fad80 ffff88007c073d20 ffff880079498000 ffff88007c073cb0 [12064.101249] Call Trace: [12064.101249] [<ffffffff816050da>] dump_stack+0x19/0x1b [12064.101249] [<ffffffff8106e34b>] warn_slowpath_common+0x6b/0xb0 [12064.101249] [<ffffffff8106e49a>] warn_slowpath_null+0x1a/0x20 [12064.101249] [<ffffffff8104158f>] native_smp_send_reschedule+0x5f/0x70 [12064.101249] [<ffffffff810a5ba4>] resched_task+0x64/0x70 [12064.101249] [<ffffffff810a6795>] check_preempt_curr+0x75/0xa0 [12064.101249] [<ffffffff810adaab>] move_task+0x4b/0x50 [12064.101249] [<ffffffff810b06c4>] active_load_balance_cpu_stop+0x1e4/0x230 [12064.101249] [<ffffffff810b04e0>] ? can_migrate_task+0x430/0x430 [12064.101249] [<ffffffff810f2b88>] cpu_stopper_thread+0x88/0x160 [12064.101249] [<ffffffff81609ef5>] ? __schedule+0x2c5/0x7b0 [12064.101249] [<ffffffff8109fdaf>] smpboot_thread_fn+0xff/0x1a0 [12064.101249] [<ffffffff8109fcb0>] ? lg_global_unlock+0xc0/0xc0 [12064.101249] [<ffffffff8109739f>] kthread+0xcf/0xe0 [12064.101249] [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 [12064.101249] [<ffffffff81614f7c>] ret_from_fork+0x7c/0xb0 [12064.101249] [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 [12064.101249] ---[ end trace 2cffddad1379cd23 ]---