Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-8644] Kernel Panic with obdfilter-survey: spl_kmem_cache_alloc+0x99/0x150 https://jira.whamcloud.com/browse/LU-8644 Lustre <p>Under obdfilter-surver system kernel panics (immediately) with SPL error; see vmcore below:</p> Lustre Master Build 3450: ZFS 0.6.5.7-1, kernel 3.10.0-327.36.1.el7_lustre.x86_64<br/> <br/> All storage devices are NVMe LU-8644

Kernel Panic with obdfilter-survey: spl_kmem_cache_alloc+0x99/0x150

Bug Major Resolved Cannot Reproduce Nathaniel Clark Adam Roe Tue, 27 Sep 2016 10:11:23 +0000 Fri, 14 Oct 2016 17:43:08 +0000 Fri, 14 Oct 2016 17:42:34 +0000 Lustre 2.8.0 Lustre 2.9.0 0 4 <p>*<b>zpool configuration & format:</b>*</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@zlfs2-oss1 ~]# zpool create OST0000 -o ashift=12 -O recordsize=1M -f /dev/nvme0n1 [root@zlfs2-oss1 ~]# mkfs.lustre --reformat --ost --backfstype=zfs --fsname=cam --mgsnode=192.168.5.21@o2ib0 --index=0 OST0000/OST0000 Permanent disk data: Target: cam:OST0000 Index: 0 Lustre FS: cam Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.5.21@o2ib mkfs_cmd = zfs create -o canmount=off -o xattr=sa OST0000/OST0000 Writing OST0000/OST0000 properties lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=cam lustre:svname=cam:OST0000 lustre:mgsnode=192.168.5.21@o2ib [root@zlfs2-oss1 ~]# mount -vvv -t lustre OST0000/OST0000 /mnt/OST0000 arg[0] = /sbin/mount.lustre arg[1] = -v arg[2] = -o arg[3] = rw arg[4] = OST0000/OST0000 arg[5] = /mnt/OST0000 source = OST0000/OST0000 (OST0000/OST0000), target = /mnt/OST0000 options = rw checking for existing Lustre data: found Writing OST0000/OST0000 properties lustre:version=1 lustre:flags=34 lustre:index=0 lustre:fsname=cam lustre:svname=cam:OST0000 lustre:mgsnode=192.168.5.21@o2ib mounting device OST0000/OST0000 at /mnt/OST0000, flags=0x1000000 options=osd=osd-zfs,,mgsnode=192.168.5.21@o2ib,virgin,update,param=mgsnode=192.168.5.21@o2ib,svname=cam-OST0000,device=OST0000/OST0000 lustre:svname=cam-OST0000 </pre> </div></div> <p><b>vmcore</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 662.214980] Modules linked in: obdecho(OE) osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_generic crypto_null libcfs(OE) xprtrdma ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs(OE) ib_umad rdma_cm ib_cm iw_cm ib_sa intel_powerclamp coretemp intel_rapl kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper iTCO_wdt iTCO_vendor_support hfi1(OE) mxm_wmi cryptd ioatdma pcspkr ib_mad ib_core ib_addr sb_edac edac_core i2c_i801 mei_me shpchp mei sg lpc_ich mfd_core ipmi_devintf ipmi_si ipmi_msghandler acpi_power_meter acpi_pad wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c raid1 sd_mod [ 662.215589] crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper nvme ttm ixgbe ahci drm mdio libahci ptp i2c_core libata pps_core dca dm_mirror dm_region_hash dm_log dm_mod zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate [ 662.215842] CPU: 29 PID: 7653 Comm: lctl Tainted: P OE ------------ 3.10.0-327.36.1.el7_lustre.x86_64 #1 [ 662.215911] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS SE5C610.86B.01.01.0018.072020161249 07/20/2016 [ 662.215978] task: ffff882021f71700 ti: ffff881ff3c98000 task.ti: ffff881ff3c98000 [ 662.216028] RIP: 0010:[<ffffffff811c1545>] [<ffffffff811c1545>] kmem_cache_alloc+0x75/0x1d0 [ 662.216092] RSP: 0018:ffff881ff3c9b9d0 EFLAGS: 00010286 [ 662.216129] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000003f8b [ 662.216176] RDX: 0000000000003f8a RSI: 00000000000042f0 RDI: ffff88103e007a00 [ 662.216222] RBP: ffff881ff3c9ba00 R08: 0000000000019580 R09: ffffffffa000b939 [ 662.216268] R10: 00000000000008df R11: 0000000180000000 R12: 00000000047546c0 [ 662.216314] R13: 00000000000042f0 R14: ffff88103e007a00 R15: ffff88103e007a00 [ 662.216361] FS: 00007f3e446be740(0000) GS:ffff88103e9c0000(0000) knlGS:0000000000000000 [ 662.216414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 662.216452] CR2: 00000000047546c0 CR3: 0000001ff3c8d000 CR4: 00000000001407e0 [ 662.216498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 662.216544] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 662.216590] Stack: [ 662.216607] ffffffffa000b939 0000000000000000 0000000000000004 ffff88103e007a00 [ 662.216667] ffff8820219bba00 0000000000000000 ffff881ff3c9ba48 ffffffffa000b939 [ 662.216730] 00000004f3c9ba48 ffff882021f71700 ffff881003c80a40 ffff882026f6e000 [ 662.216795] Call Trace: [ 662.216839] [<ffffffffa000b939>] ? spl_kmem_cache_alloc+0x99/0x150 [spl] [ 662.216895] [<ffffffffa000b939>] spl_kmem_cache_alloc+0x99/0x150 [spl] [ 662.216999] [<ffffffffa00bbc70>] arc_buf_alloc+0x90/0x190 [zfs] [ 662.217067] [<ffffffffa00bbd8b>] arc_loan_buf+0x1b/0x30 [zfs] [ 662.217142] [<ffffffffa00cba09>] dmu_request_arcbuf+0x19/0x20 [zfs] [ 662.217204] [<ffffffffa1160901>] osd_bufs_get+0x591/0xba0 [osd_zfs] [ 662.217249] [<ffffffff811c25d3>] ? __kmalloc+0x1f3/0x230 [ 662.217313] [<ffffffffa125b78d>] ofd_preprw_write.isra.29+0x1bd/0xcd0 [ofd] [ 662.217369] [<ffffffffa125ca8a>] ofd_preprw+0x7ea/0x10c0 [ofd] [ 662.217421] [<ffffffffa1119894>] echo_client_prep_commit.isra.49+0x334/0xc30 [obdecho] [ 662.217549] [<ffffffffa0ce131f>] ? lu_object_find_slice+0x1f/0x90 [obdclass] [ 662.219311] [<ffffffffa11230af>] echo_client_iocontrol+0x9bf/0x1c40 [obdecho] [ 662.221071] [<ffffffff811c25d3>] ? __kmalloc+0x1f3/0x230 [ 662.222855] [<ffffffffa0caa15e>] class_handle_ioctl+0x19de/0x2150 [obdclass] [ 662.224622] [<ffffffff81197580>] ? handle_mm_fault+0x5e0/0xf80 [ 662.226363] [<ffffffff81285868>] ? security_capable+0x18/0x20 [ 662.228099] [<ffffffffa0c8e712>] obd_class_ioctl+0xd2/0x170 [obdclass] [ 662.229809] [<ffffffff811f2665>] do_vfs_ioctl+0x2e5/0x4c0 [ 662.231518] [<ffffffff8164215d>] ? __do_page_fault+0x16d/0x450 [ 662.233191] [<ffffffff811f28e1>] SyS_ioctl+0xa1/0xc0 [ 662.234815] [<ffffffff81646c49>] system_call_fastpath+0x16/0x1b [ 662.236410] Code: ce 00 00 49 8b 50 08 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 1f 01 00 00 48 85 c0 0f 84 16 01 00 00 49 63 46 20 48 8d 4a 01 4d 8b 06 <49> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 662.239730] RIP [<ffffffff811c1545>] kmem_cache_alloc+0x75/0x1d0 [ 662.241291] RSP <ffff881ff3c9b9d0> [ 662.242792] CR2: 00000000047546c0 </pre> </div></div> <p><b>zfs.conf</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options zfs metaslab_debug_unload=1 options zfs zfs_vdev_scheduler=deadline options zfs zfs_arc_max=103079215104 options zfs zfs_dirty_data_max=4294967296 options zfs zfs_vdev_async_write_active_min_dirty_percent=20 options zfs zfs_vdev_async_write_min_active=5 options zfs zfs_vdev_async_write_max_active=10 options zfs zfs_vdev_sync_read_min_active=16 options zfs zfs_vdev_sync_read_max_active=16 </pre> </div></div> <p>Normal workloads (IOR etc) when mounted are fine. I have only observed the issue when using obdfilter-survey.</p> <p>Adam, <br/> Do you have the full console log or messages file from the machine?</p> <p>I don't I'm afraid - this system has since had its base OS reprovisioned. </p> <p>If this recurs, please capture full console log and this bug can be reopened.</p> Development Rank 1|hzyppb: Rank (Obsolete) 9223372036854775807 Severity