Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-13596] usercopy: kernel memory exposure attempt detected from ffff98c06ba17d80 (kmalloc-128) (48032 bytes) https://jira.whamcloud.com/browse/LU-13596 Lustre <p>Our Lustre 2.10 stack is here: <a href="https://github.com/LLNL/lustre/releases/tag/2.10.8_9.chaos" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/releases/tag/2.10.8_9.chaos</a></p> <p>Compute node crashes with an LBUG. dmesg-vmcore.txt contains</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[677308.381183] usercopy: kernel memory exposure attempt detected from ffff98c06ba17d80 (kmalloc-128) (48032 bytes) [677308.382142] usercopy: kernel memory exposure attempt detected from ffff98b452b38d80 (kmalloc-128) (48032 bytes) [677308.382152] usercopy: kernel memory exposure attempt detected from ffff98b485643c00 (kmalloc-128) (48032 bytes) [677308.382156] usercopy: kernel memory exposure attempt detected from ffff98acb7964b80 (kmalloc-128) (48032 bytes) [677308.382192] ------------[ cut here ]------------ [677308.382193] kernel BUG at mm/usercopy.c:72! [677308.382195] invalid opcode: 0000 [#1] SMP [677308.382230] Modules linked in: osc(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) bonding rpcrdma ib_iser opa_vnic iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi hfi1 kvm ocrdma(T) irqbypass pcspkr rdmavt joydev sg lpc_ich i2c_i801 ioatdma ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_owner xfs xt_conntrack nf_conntrack libcrc32c iptable_filter acpi_cpufreq ib_ipoib sch_fq_codel rdma_ucm ib_uverbs binfmt_misc ib_umad msr_safe(OE) iw_cxgb4 rdma_cm iw_cm ib_cm iw_cxgb3 ib_core ip_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) ext4 mbcache jbd2 dm_service_time [677308.382254] be2iscsi sd_mod crc_t10dif crct10dif_generic bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi 8021q libcxgb garp mrp stp qla4xxx llc iscsi_boot_sysfs mgag200 drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea crc32_pclmul sysfillrect sysimgblt crc32c_intel fb_sys_fops ghash_clmulni_intel igb ttm aesni_intel ahci lrw drm mxm_wmi gf128mul libahci glue_helper dca ablk_helper ptp cryptd libata be2net drm_panel_orientation_quirks pps_core i2c_algo_bit dm_multipath wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi [677308.382257] CPU: 30 PID: 46949 Comm: VCPoissonSolve3 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1127.0.0.1chaos.ch6.x86_64 #1 [677308.382258] Hardware name: Penguin Computing Relion 2900e/S2600WT2R, BIOS SE5C610.86B.01.01.0027.071020182329 07/10/2018 [677308.382259] task: ffff98b45c04d230 ti: ffff98bbb56f0000 task.ti: ffff98bbb56f0000 [677308.382266] RIP: 0010:[&lt;ffffffffa805bc17&gt;] [&lt;ffffffffa805bc17&gt;] __check_object_size+0x87/0x250 [677308.382267] RSP: 0018:ffff98bbb56f3c50 EFLAGS: 00010246 [677308.382268] RAX: 0000000000000063 RBX: ffff98b485643c00 RCX: 0000000000000000 [677308.382269] RDX: 0000000000000000 RSI: 0000000000000292 RDI: 0000000000000292 [677308.382270] RBP: ffff98bbb56f3c70 R08: ffffffffa8e0387c R09: ffffffffa8e75bc7 [677308.382271] R10: 000000000008dd28 R11: 0000000000100000 R12: 000000000000bba0 [677308.382271] R13: 0000000000000001 R14: ffff98b48564f7a0 R15: 000000000000bba0 [677308.382273] FS: 00002aaaaab1bb40(0000) GS:ffff98c3bef00000(0000) knlGS:0000000000000000 [677308.382274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [677308.382274] CR2: 000000000099d000 CR3: 0000001332614000 CR4: 00000000003607e0 [677308.382275] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [677308.382276] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [677308.382277] Call Trace: [677308.382290] [&lt;ffffffffc0cfe245&gt;] lov_getstripe+0x705/0x9b0 [lov] [677308.382296] [&lt;ffffffffa804fa42&gt;] ? __mem_cgroup_commit_charge+0x112/0x340 [677308.382301] [&lt;ffffffffc0cfc64f&gt;] lov_object_getstripe+0x6f/0x180 [lov] [677308.382336] [&lt;ffffffffc122f55e&gt;] cl_object_getstripe+0x6e/0x130 [obdclass] [677308.382352] [&lt;ffffffffc13bae20&gt;] ll_file_getstripe+0x70/0x170 [lustre] [677308.382361] [&lt;ffffffffc13d0530&gt;] ll_file_ioctl+0x11b0/0x3830 [lustre] [677308.382364] [&lt;ffffffffa8004674&gt;] ? handle_mm_fault+0x3a4/0x9b0 [677308.382367] [&lt;ffffffffa8075600&gt;] do_vfs_ioctl+0x420/0x6d0 [677308.382370] [&lt;ffffffffa85ba76b&gt;] ? __do_page_fault+0x24b/0x550 [677308.382372] [&lt;ffffffffa8075951&gt;] SyS_ioctl+0xa1/0xc0 [677308.382374] [&lt;ffffffffa85c0112&gt;] system_call_fastpath+0x25/0x2a [677308.382392] Code: 45 d1 48 c7 c6 62 80 88 a8 48 c7 c1 4b 19 89 a8 48 0f 45 f1 49 89 c0 4d 89 e1 48 89 d9 48 c7 c7 d8 e6 88 a8 31 c0 e8 66 a7 54 00 &lt;0f&gt; 0b 0f 1f 80 00 00 00 00 48 c7 c0 00 00 e0 a7 4c 39 f0 73 0d [677308.382394] RIP [&lt;ffffffffa805bc17&gt;] __check_object_size+0x87/0x250 [677308.382395] RSP &lt;ffff98bbb56f3c50&gt; </pre> </div></div> kernel 3.10.0-1127.0.0.1chaos.ch6.x86_64 <br/> lustre 2.10.8_9.chaos LU-13596 usercopy: kernel memory exposure attempt detected from ffff98c06ba17d80 (kmalloc-128) (48032 bytes) Bug Minor Resolved Duplicate Peter Jones Olaf Faaland Fri, 22 May 2020 22:47:14 +0000 Mon, 1 Jun 2020 16:57:42 +0000 Fri, 29 May 2020 22:39:13 +0000 Lustre 2.10.8 0 3 <p>Similar (same kernel protection mechanism) to <a href="https://jira.whamcloud.com/browse/LU-12331" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-12331</a></p> <p>I'm unable to reproduce this so far.<br/> But we had 10 nodes hit this yesterday, in a 3000-node cluster. Before yesterday, I believe we've not seen this.</p> <p>We recently updated to RHEL 7.8 based TOSS (about 9 days ago for this cluster)</p> <p>For my recordkeeping, my local ticket is TOSS4802</p> <p>We intend to update this cluster to Lustre 2.12 fairly soon, so I hesitate to backport any patches to our 2.10 stack. However, if we see this issue at the same rate we saw it yesterday, I may be forced to, depending on the complexity of the patch.</p> <p>Either way, I would like to understand why we're seeing this now, if possible. And I would like to determine whether 2.12 has the issue.</p> <p>It looks like this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-12580" title="usercopy exposure attempt detected in LL_IOC_LOV_GETSTRIPE ioctl" class="issue-link" data-issue-key="LU-12580"><del>LU-12580</del></a> "<tt>usercopy exposure attempt detected in LL_IOC_LOV_GETSTRIPE ioctl</tt>" and is fixed by patch <a href="https://review.whamcloud.com/38050" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38050</a><br/> "<tt><a href="https://jira.whamcloud.com/browse/LU-12580" title="usercopy exposure attempt detected in LL_IOC_LOV_GETSTRIPE ioctl" class="issue-link" data-issue-key="LU-12580"><del>LU-12580</del></a> lov: fix typo in lov_comp_md_size</tt>" and patch <a href="https://review.whamcloud.com/38051" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38051</a><br/> "<tt><a href="https://jira.whamcloud.com/browse/LU-12580" title="usercopy exposure attempt detected in LL_IOC_LOV_GETSTRIPE ioctl" class="issue-link" data-issue-key="LU-12580"><del>LU-12580</del></a> lov: fix out of bound usercopy</tt>".</p> <p>I agree, it looks like a dupe of <a href="https://jira.whamcloud.com/browse/LU-12580" title="usercopy exposure attempt detected in LL_IOC_LOV_GETSTRIPE ioctl" class="issue-link" data-issue-key="LU-12580"><del>LU-12580</del></a>. Sorry, not sure how I missed that.</p> <p>NP. Should be fixed in 2.12.5 then.</p> Duplicate LU-12580 Development Rank 1|i01153: Rank (Obsolete) 9223372036854775807 Severity