[LU-8689] sanity test_900: soft lockup - CPU#0 stuck for 67s! [mount.lustre:28303]

Tue, 6 Jul 2021 19:04:40 +0000

This issue was created by maloo for Bob Glossman <bob.glossman@intel.com>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/2ce61b88-8f31-11e6-a9b0-5254006e85c2.

The sub-test test_900 failed with the following error:

test failed to respond and timed out

TIMEOUT seen during mount command on zfs
from OST console log:

09:53:54:BUG: soft lockup - CPU#0 stuck for 67s! [mount.lustre:28303]
09:53:54:Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic libcfs(U) nfsd exportfs autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
09:53:54:CPU 0 
09:53:54:Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic libcfs(U) nfsd exportfs autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
09:53:54:
09:53:54:Pid: 28303, comm: mount.lustre Tainted: P           -- ------------    2.6.32-573.12.1.el6_lustre.g99dd2a1.x86_64 #1 Red Hat KVM
09:53:54:RIP: 0010:[<ffffffff8129e2a9>]  [<ffffffff8129e2a9>] __write_lock_failed+0x9/0x20
09:53:54:RSP: 0018:ffff8800787eb8a0  EFLAGS: 00000287
09:53:54:RAX: 0000000000000000 RBX: ffff8800787eb8a8 RCX: ffff88002cc0abd8
09:53:54:RDX: 0000000000000000 RSI: ffff880018e3e000 RDI: ffff880018e3e0b8
09:53:54:RBP: ffffffff8100bc0e R08: dead000000200200 R09: dead000000100100
09:53:54:R10: dead000000200200 R11: 0000000000000000 R12: ffff880000000010
09:53:54:R13: 0000000000008050 R14: ffffffffa0210f2f R15: ffff8800787eb858
09:53:54:FS:  00007f012b84d7a0(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
09:53:54:CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
09:53:54:CR2: 00007f012ad6d0c0 CR3: 00000000597cc000 CR4: 00000000000006f0
09:53:54:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
09:53:54:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
09:53:54:Process mount.lustre (pid: 28303, threadinfo ffff8800787e8000, task ffff880051a10ab0)
09:53:54:Stack:
09:53:54: ffffffff8153c697 ffff8800787eb8e8 ffffffffa0f0cf85 ffff8800787eb908
09:53:54:<d> ffff880018e3e000 ffff8800787eb958 ffff8800787eb958 ffff88002cc0a940
09:53:54:<d> ffff8800787eb958 ffff8800787eb908 ffffffffa0f0004c ffff880018e3e000
09:53:54:Call Trace:
09:53:54: [<ffffffff8153c697>] ? _write_lock+0x17/0x20
09:53:54: [<ffffffffa0f0cf85>] ? osd_oi_fini+0x45/0x4a0 [osd_zfs]
09:53:54: [<ffffffffa0f0004c>] ? osd_device_fini+0x12c/0x530 [osd_zfs]
09:53:54: [<ffffffffa0f006d0>] ? osd_device_alloc+0x280/0x290 [osd_zfs]
09:53:54: [<ffffffffa089a05f>] ? obd_setup+0x1bf/0x290 [obdclass]
09:53:54: [<ffffffffa089a389>] ? class_setup+0x259/0x930 [obdclass]
09:53:54: [<ffffffffa08a0c01>] ? class_process_config+0x1151/0x24c0 [obdclass]
09:53:54: [<ffffffff81178a2c>] ? __kmalloc+0x21c/0x230
09:53:54: [<ffffffffa08a946f>] ? do_lcfg+0x2cf/0x9f0 [obdclass]
09:53:54: [<ffffffffa08a9c24>] ? lustre_start_simple+0x94/0x200 [obdclass]
09:53:54: [<ffffffffa08d433f>] ? osd_start+0x54f/0x840 [obdclass]
09:53:54: [<ffffffff8129c28c>] ? simple_strtoull+0x2c/0x50
09:53:54: [<ffffffff8129c3a9>] ? simple_strtoul+0x9/0x10
09:53:54: [<ffffffffa08a56e8>] ? target_name2index+0x78/0xc0 [obdclass]
09:53:54: [<ffffffffa08d96a3>] ? server_fill_super+0x253/0x1450 [obdclass]
09:53:54: [<ffffffffa07a5cf1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
09:53:54: [<ffffffffa08abd48>] ? lustre_fill_super+0x348/0x8c0 [obdclass]
09:53:54: [<ffffffffa08aba00>] ? lustre_fill_super+0x0/0x8c0 [obdclass]
09:53:54: [<ffffffff8119542f>] ? get_sb_nodev+0x5f/0xa0
09:53:54: [<ffffffffa08a5665>] ? lustre_get_sb+0x25/0x30 [obdclass]
09:53:54: [<ffffffff81194a6b>] ? vfs_kern_mount+0x7b/0x1b0
09:53:54: [<ffffffff81194c12>] ? do_kern_mount+0x52/0x130
09:53:54: [<ffffffff811a7972>] ? vfs_ioctl+0x22/0xa0
09:53:54: [<ffffffff811b6bdb>] ? do_mount+0x2fb/0x930
09:53:54: [<ffffffff811b72a0>] ? sys_mount+0x90/0xe0
09:53:54: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
09:53:54:Code: 00 00 48 8b 5b 20 48 83 eb 07 48 39 d9 73 06 48 89 01 31 c0 c3 b8 f2 ff ff ff c3 90 90 90 90 90 90 90 f0 81 07 00 00 00 01 f3 90 <81> 3f 00 00 00 01 75 f6 f0 81 2f 00 00 00 01 0f 85 e2 ff ff ff

Info required for matching: sanity 900

Whamcloud Community JIRA

[LU-8689] sanity test_900: soft lockup - CPU#0 stuck for 67s! [mount.lustre:28303]