[LU-8327] conf_sanity test_61: soft lockup

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.9.0
• Severity: 3
• Assignee: Yang Sheng
• Reporter: nasf (Inactive)

Description

The logs on the OST:

07:05:17:[25848.056002] BUG: soft lockup - CPU#0 stuck for 22s! [ll_ost_seq00_00:30636]
07:05:17:[25848.056002] Modules linked in: lustre(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) obdecho(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) ptlrpc_gss(OE) osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) ldiskfs(OE) sha512_generic crypto_null dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xprtrdma ppdev pcspkr ib_isert virtio_balloon iscsi_target_mod i2c_piix4 parport_pc parport ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic crct10dif_common ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk cirrus syscopyarea sysfillrect sysimgblt drm_kms_helper ttm 8139too ata_piix drm 8139cp serio_raw mii virtio_pci virtio_ring i2c_core virtio libata floppy [last unloaded: libcfs]
07:05:17:[25848.056002] CPU: 0 PID: 30636 Comm: ll_ost_seq00_00 Tainted: G           OE  ------------   3.10.0-327.18.2.el7_lustre.x86_64 #1
07:05:17:[25848.056002] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
07:05:17:[25848.056002] task: ffff880047004500 ti: ffff8800546cc000 task.ti: ffff8800546cc000
07:05:17:[25848.056002] RIP: 0010:[<ffffffff8163d5f2>]  [<ffffffff8163d5f2>] _raw_spin_lock+0x32/0x50
07:05:17:[25848.056002] RSP: 0018:ffff8800546cf850  EFLAGS: 00000293
07:05:17:[25848.056002] RAX: 0000000000000fb8 RBX: ffff880079f5fea0 RCX: 0000000000000bde
07:05:17:[25848.056002] RDX: 0000000000000b24 RSI: 0000000000000b24 RDI: ffff8800478e5ba0
07:05:17:[25848.056002] RBP: ffff8800546cf850 R08: 4010000000000000 R09: 0079f5fea0080000
07:05:17:[25848.056002] R10: ff680a1fdd77a802 R11: 000000000000000f R12: ffff88007bb5c340
07:05:17:[25848.056002] R13: ffff880079f5fea0 R14: 00000000659b4173 R15: ffff88004b4648f0
07:05:17:[25848.056002] FS:  0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
07:05:17:[25848.056002] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
07:05:17:[25848.056002] CR2: 00007f4a800054a9 CR3: 000000000194a000 CR4: 00000000000006f0
07:05:17:[25848.056002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
07:05:17:[25848.056002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
07:05:17:[25848.056002] Stack:
07:05:17:[25848.056002]  ffff8800546cf8d8 ffffffffa015cbfc ffffffffa0698656 01ff88007922ff38
07:05:17:[25848.056002]  0000000000008603 ffff8800478e5800 0000000000000000 00000000659b4173
07:05:17:[25848.056002]  ffff8800546cf904 0000000000000001 ffff8800546cf940 00000000659b4173
07:05:17:[25848.056002] Call Trace:
07:05:17:[25848.056002]  [<ffffffffa015cbfc>] do_get_write_access+0x32c/0x4e0 [jbd2]
07:05:17:[25848.056002]  [<ffffffffa0698656>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
07:05:17:[25848.056002]  [<ffffffffa015cdd7>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
07:05:17:[25848.056002]  [<ffffffffa064670b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
07:05:17:[25848.056002]  [<ffffffffa0c2a108>] iam_txn_add+0x28/0x60 [osd_ldiskfs]
07:05:17:[25848.056002]  [<ffffffffa0c2c953>] iam_add_rec+0x43/0x2e0 [osd_ldiskfs]
07:05:17:[25848.056002]  [<ffffffffa0c2b29e>] ? __iam_it_get+0x1ae/0x1b0 [osd_ldiskfs]
07:05:17:[25848.056002]  [<ffffffffa0c2d49e>] iam_insert+0xce/0x120 [osd_ldiskfs]
07:05:17:[25848.056002]  [<ffffffffa0c18925>] osd_index_iam_insert+0x225/0x530 [osd_ldiskfs]
07:05:17:[25848.056002]  [<ffffffffa0967388>] fld_index_create+0x1d8/0x7a0 [fld]
07:05:17:[25848.056002]  [<ffffffffa05e8654>] ? libcfs_log_return+0x24/0x30 [libcfs]
07:05:17:[25848.056002]  [<ffffffffa0967bb1>] fld_insert_entry+0x261/0x370 [fld]
07:05:17:[25848.056002]  [<ffffffffa097c9d1>] seq_server_check_and_alloc_super+0x1c1/0x310 [fid]
07:05:17:[25848.056002]  [<ffffffffa097cb8f>] seq_server_alloc_meta+0x6f/0x5d0 [fid]
07:05:17:[25848.056002]  [<ffffffffa097df85>] seq_handler+0x235/0x4b0 [fid]
07:05:17:[25848.056002]  [<ffffffffa0a61f25>] tgt_request_handle+0x915/0x1320 [ptlrpc]
07:05:17:[25848.056002]  [<ffffffffa0a0e4bb>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]
07:05:17:[25848.056002]  [<ffffffffa0a0c078>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
07:05:17:[25848.056002]  [<ffffffffa05eb957>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
07:05:17:[25848.056002]  [<ffffffffa0a12570>] ptlrpc_main+0xaa0/0x1dd0 [ptlrpc]
07:05:17:[25848.056002]  [<ffffffff81013588>] ? __switch_to+0xf8/0x4b0
07:05:17:[25848.056002]  [<ffffffffa0a11ad0>] ? ptlrpc_register_service+0xe40/0xe40 [ptlrpc]
07:05:17:[25848.056002]  [<ffffffff810a5acf>] kthread+0xcf/0xe0
07:05:17:[25848.056002]  [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
07:05:17:[25848.056002]  [<ffffffff81646318>] ret_from_fork+0x58/0x90
07:05:17:[25848.056002]  [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
07:05:17:[25848.056002] Code: 89 e5 b8 00 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 eb 0c 0f 1f 44 00 00 f3 90 <83> e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 0f 1f 80 00 00 00 

Issue Links

• is related to: LU-8332 sanity-hsm soft lock up in journal write
• is related to: LU-8685 Fix JBD2 issue in EL7 Kernels