[LU-12908] sanity-scrub: BUG: soft lockup - CPU#1 stuck for 22s! [OI_scrub:24284]

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for wangshilong <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/9fb4e414-f7a7-11e9-be86-52540065bddc

7443.568005] LustreError: 24365:0:(osd_index.c:218:__osd_xattr_load_by_oid()) lustre-MDT0002: can't get bonus, rc = -17
[ 7443.570046] LustreError: 24365:0:(osd_index.c:218:__osd_xattr_load_by_oid()) Skipped 3 previous similar messages
[ 7443.659719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub |
awk '/^status/

{ print $2 }

'
[ 7468.310268] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [lfsck:24283]
[ 7468.316709] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [OI_scrub:24284]
[ 7468.316807] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core sunrpc dm_mod zfs(POE) zunicode(POE) zavl(POE) icp(POE) iosf_mbi crc32_pclmul zcommon(POE) znvpair(POE) spl(OE) ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper parport_pc ablk_helper cryptd joydev parport virtio_balloon pcspkr i2c_piix4 ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk ata_piix
[ 7468.316822] libata crct10dif_pclmul crct10dif_common 8139too virtio_pci crc32c_intel serio_raw virtio_ring virtio floppy 8139cp mii
[ 7468.316857] CPU: 1 PID: 24284 Comm: OI_scrub Kdump: loaded Tainted: P OE ------------ 3.10.0-957.27.2.el7_lustre.ddn1.x86_64 #1
[ 7468.316858] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 7468.316863] task: ffff967e13aa8000 ti: ffff967e1f110000 task.ti: ffff967e1f110000
[ 7468.316919] RIP: 0010:[<ffffffff8ef6cf9b>] [<ffffffff8ef6cf9b>] native_safe_halt+0xb/0x20
[ 7468.316924] RSP: 0018:ffff967e1f113d78 EFLAGS: 00000202
[ 7468.316925] RAX: 0000000000000001 RBX: ffff967e1f113dc0 RCX: 0000000000000001
[ 7468.316925] RDX: 0000000000000002 RSI: 0000000000000001 RDI: ffff967e3fd1b7c4
[ 7468.316926] RBP: ffff967e1f113d78 R08: 000000000000000a R09: 000000000000fffe
[ 7468.316927] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000020
[ 7468.316928] R13: ffff967e1f113df0 R14: ffff967e1d942000 R15: ffff967e1d9432d8
[ 7468.316932] FS: 0000000000000000(0000) GS:ffff967e3fd00000(0000) knlGS:0000000000000000
[ 7468.316934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7468.316934] CR2: 00007fb9588a6e00 CR3: 000000007abf2000 CR4: 00000000000606e0
[ 7468.316948] Call Trace:
[ 7468.316998] [<ffffffff8e86a9e5>] kvm_wait+0x65/0x70
[ 7468.317021] [<ffffffff8e913346>] __pv_queued_spin_lock_slowpath+0x216/0x2e0
[ 7468.317030] [<ffffffff8ef5f2cb>] queued_spin_lock_slowpath+0xb/0xf
[ 7468.317033] [<ffffffff8ef6d7a0>] _raw_spin_lock+0x20/0x30
[ 7468.317118] [<ffffffffc131f470>] osd_scrub_main+0x720/0xf50 [osd_zfs]
[ 7468.317135] [<ffffffff8e8d7c40>] ? wake_up_state+0x20/0x20
[ 7468.317144] [<ffffffffc131ed50>] ? osd_scrub_exec+0x960/0x960 [osd_zfs]
[ 7468.317156] [<ffffffff8e8c2e81>] kthread+0xd1/0xe0
[ 7468.317158] [<ffffffff8e8c2db0>] ? insert_kthread_work+0x40/0x40
[ 7468.317165] [<ffffffff8ef77c37>] ret_from_fork_nospec_begin+0x21/0x21
[ 7468.317166] [<ffffffff8e8c2db0>] ? insert_kthread_work+0x40/0x40
[ 7468.317199] Code: 30 e9 7a ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 00 2d d5 af 09 00 e9 3d ff ff ff 0f 1f 40 00 55 48 89 e5 66 66 66 66 90 fb f4 <5d> c3 0f 1f 00 0f 00 2d 29 bf 09 00 eb f0 0f 1f 80 00 00 00 00