[LU-7246] sanity test_900: list_del corruption

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Minor
• Affects Version/s: Lustre 2.8.0
• Labels: zfs
• Environment: client and server: lustre-master build #3192 RHEL6.7 zfs
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for sarah_lw <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/8fc2cf88-6158-11e5-b495-5254006e85c2.

The sub-test test_900 failed with the following error:

test failed to respond and timed out

oss console

10:41:30:Lustre: 8465:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
10:41:30:Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST '
10:41:30:Lustre: DEBUG MARKER: ! zpool list -H lustre-ost3 >/dev/null 2>&1 ||
10:41:30:			grep -q ^lustre-ost3/ /proc/mounts ||
10:41:30:			zpool export  lustre-ost3
10:41:30:------------[ cut here ]------------
10:41:30:WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: P           -- ------------   )
10:41:30:Hardware name: KVM
10:41:30:list_del corruption. prev->next should be ffffc900084aa010, but was 9117e7e5b595839c
10:41:30:Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) microcode serio_raw virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
10:41:30:Pid: 8573, comm: zpool Tainted: P           -- ------------    2.6.32-573.3.1.el6_lustre.gc3d2503.x86_64 #1
10:41:30:Call Trace:
10:41:30: [<ffffffff81077491>] ? warn_slowpath_common+0x91/0xe0
10:41:30: [<ffffffff81077596>] ? warn_slowpath_fmt+0x46/0x60
10:41:30: [<ffffffff812a40ae>] ? list_del+0x6e/0xa0
10:41:30: [<ffffffffa019fc20>] ? __spl_cache_flush+0x140/0x160 [spl]
10:41:30: [<ffffffffa019fc86>] ? spl_cache_flush+0x46/0x70 [spl]
10:41:30: [<ffffffffa01a1785>] ? spl_kmem_cache_free+0x1f5/0x210 [spl]
10:41:30: [<ffffffffa02a56a4>] ? zio_data_buf_free+0x24/0x30 [zfs]
10:41:30: [<ffffffffa01f7de2>] ? arc_buf_data_free+0x22/0x50 [zfs]
10:41:30: [<ffffffffa01f83a5>] ? arc_buf_destroy+0x145/0x1c0 [zfs]
10:41:30: [<ffffffffa01fd945>] ? arc_evict+0x5e5/0x930 [zfs]
10:41:30: [<ffffffffa01fe6fa>] ? arc_flush+0x3a/0x1a0 [zfs]
10:41:30: [<ffffffffa0233098>] ? dsl_pool_close+0xb8/0x140 [zfs]
10:41:30: [<ffffffffa024f2fc>] ? spa_unload+0xdc/0x270 [zfs]
10:41:30: [<ffffffffa024f802>] ? spa_export_common+0x262/0x3f0 [zfs]
10:41:30: [<ffffffffa019f6db>] ? spl_kmem_free+0x2b/0x40 [spl]
10:41:30: [<ffffffffa024f9cb>] ? spa_export+0x1b/0x20 [zfs]
10:41:30: [<ffffffffa028560f>] ? zfs_ioc_pool_export+0x3f/0x60 [zfs]
10:41:30: [<ffffffffa019f699>] ? strdup+0x69/0x80 [spl]
10:41:30: [<ffffffffa0287955>] ? zfsdev_ioctl+0x495/0x4d0 [zfs]
10:41:30: [<ffffffff811a7802>] ? vfs_ioctl+0x22/0xa0
10:41:30: [<ffffffff811a79a4>] ? do_vfs_ioctl+0x84/0x580
10:41:30: [<ffffffff811a7f21>] ? sys_ioctl+0x81/0xa0
10:41:30: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
10:41:30:---[ end trace 1d04c76a3e7b1bf8 ]---
10:41:30:general protection fault: 0000 [#1] SMP 
10:41:30:last sysfs file: /sys/devices/system/cpu/online
10:41:30:CPU 0 
10:41:30:Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) microcode serio_raw virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
10:41:30:
10:41:30:Pid: 8573, comm: zpool Tainted: P        W  -- ------------    2.6.32-573.3.1.el6_lustre.gc3d2503.x86_64 #1 Red Hat KVM
10:41:30:RIP: 0010:[<ffffffff812a40f7>]  [<ffffffff812a40f7>] __list_add+0x17/0xa0
10:41:30:RSP: 0018:ffff88004c5ab928  EFLAGS: 00010086
10:41:30:RAX: 00000000fffe0000 RBX: ffffc900087ca018 RCX: ffff8800693c1520
10:41:30:RDX: aa915d4fcb7b2895 RSI: ffffc900086c2020 RDI: ffffc900087ca018
10:41:30:RBP: ffff88004c5ab948 R08: ffff88007be38098 R09: 0000000000000000
10:41:30:R10: ffff880062f98d98 R11: ffff880062f98d40 R12: ffff88007be30000
10:41:30:R13: 0000000000000005 R14: ffff8800794c8e28 R15: 0000000000000008
10:41:30:FS:  00007fd09f742b40(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
10:41:30:CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
10:41:30:CR2: 00007ffe10c30000 CR3: 000000006a4e1000 CR4: 00000000000006f0
10:41:30:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
10:41:30:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
10:41:30:Process zpool (pid: 8573, threadinfo ffff88004c5a8000, task ffff8800693c1520)
10:41:30:Stack:
10:41:30: ffff88004c5ab948 ffffc900086c2000 ffff88007be30000 0000000000000005
10:41:30:<d> ffff88004c5ab9a8 ffffffffa019fb6c ffff880037b7eae8 ffff8800373406e0
10:41:30:<d> ffff88007be38098 ffff8800794c8e00 ffff8800693c1558 ffff88007be380b0
10:41:30:Call Trace:
10:41:30: [<ffffffffa019fb6c>] __spl_cache_flush+0x8c/0x160 [spl]
10:41:30: [<ffffffffa019fc86>] spl_cache_flush+0x46/0x70 [spl]
10:41:30: [<ffffffffa01a1785>] spl_kmem_cache_free+0x1f5/0x210 [spl]
10:41:30: [<ffffffffa02a56a4>] zio_data_buf_free+0x24/0x30 [zfs]
10:41:30: [<ffffffffa01f7de2>] arc_buf_data_free+0x22/0x50 [zfs]
10:41:30: [<ffffffffa01f83a5>] arc_buf_destroy+0x145/0x1c0 [zfs]
10:41:30: [<ffffffffa01fd945>] arc_evict+0x5e5/0x930 [zfs]
10:41:30: [<ffffffffa01fe6fa>] arc_flush+0x3a/0x1a0 [zfs]
10:41:30: [<ffffffffa0233098>] dsl_pool_close+0xb8/0x140 [zfs]
10:41:30: [<ffffffffa024f2fc>] spa_unload+0xdc/0x270 [zfs]
10:41:30: [<ffffffffa024f802>] spa_export_common+0x262/0x3f0 [zfs]
10:41:30: [<ffffffffa019f6db>] ? spl_kmem_free+0x2b/0x40 [spl]
10:41:30: [<ffffffffa024f9cb>] spa_export+0x1b/0x20 [zfs]
10:41:30: [<ffffffffa028560f>] zfs_ioc_pool_export+0x3f/0x60 [zfs]
10:41:30: [<ffffffffa019f699>] ? strdup+0x69/0x80 [spl]
10:41:30: [<ffffffffa0287955>] zfsdev_ioctl+0x495/0x4d0 [zfs]
10:41:30: [<ffffffff811a7802>] vfs_ioctl+0x22/0xa0
10:41:30: [<ffffffff811a79a4>] do_vfs_ioctl+0x84/0x580
10:41:30: [<ffffffff811a7f21>] sys_ioctl+0x81/0xa0
10:41:30: [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
10:41:30:Code: ff 48 8b 03 eb 92 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <4c> 8b 42 08 49 89 f5 49 89 d4 49 39 f0 75 27 4d 8b 45 00 4d 39 
10:41:30:RIP  [<ffffffff812a40f7>] __list_add+0x17/0xa0
10:41:30: RSP <ffff88004c5ab928>
10:41:30:Initializing cgroup subsys cpuset