https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11204 Lustre <p>Seeing these for some time in my testing now, in racer:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[48792.659356] BUG: unable to handle kernel paging request at ffff88008278be60 [48792.659356] IP: [&lt;ffffffffa034f110&gt;] lu_object_put+0x270/0x3c0 [obdclass] [48792.659356] PGD 23e3067 PUD 33fa01067 PMD 33f9ed067 PTE 800000008278b060 [48792.659356] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [48792.659356] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc_t10dif crct10dif_generic crct10dif_common ata_generic ttm pata_acpi drm_kms_helper i2c_piix4 ata_piix drm virtio_balloon pcspkr serio_raw virtio_console virtio_blk i2c_core libata floppy ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [48792.686829] CPU: 1 PID: 21888 Comm: mdt00_002 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.5-debug #1 [48792.686829] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [48792.686829] task: ffff88009d644c80 ti: ffff8800b93ac000 task.ti: ffff8800b93ac000 [48792.686829] RIP: 0010:[&lt;ffffffffa034f110&gt;] [&lt;ffffffffa034f110&gt;] lu_object_put+0x270/0x3c0 [obdclass] [48792.686829] RSP: 0018:ffff8800b93afb38 EFLAGS: 00010246 [48792.686829] RAX: 0000000000000000 RBX: ffff88030ef74160 RCX: 0000000000000002 [48792.686829] RDX: 0000000000000002 RSI: ffffc90007768000 RDI: ffff88008278be68 [48792.686829] RBP: ffff8800b93afb88 R08: 00000000000000cc R09: 000000000000004f [48792.686829] R10: 0000000000000b01 R11: 00000000003fffff R12: ffff880291d79600 [48792.686829] R13: ffff88008278bea0 R14: ffff88008278be50 R15: ffffc900077a8028 [48792.686829] FS: 0000000000000000(0000) GS:ffff88033da40000(0000) knlGS:0000000000000000 [48792.686829] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [48792.686829] CR2: ffff88008278be60 CR3: 000000024c172000 CR4: 00000000000006e0 [48792.686829] Call Trace: [48792.686829] [&lt;ffffffffa0cbbb13&gt;] mdt_reint_unlink+0x7c3/0x1410 [mdt] [48792.686829] [&lt;ffffffffa0cbfc10&gt;] mdt_reint_rec+0x80/0x210 [mdt] [48792.686829] [&lt;ffffffffa0c9f6ab&gt;] mdt_reint_internal+0x5fb/0x990 [mdt] [48792.686829] [&lt;ffffffffa0caa4a7&gt;] mdt_reint+0x67/0x140 [mdt] [48792.686829] [&lt;ffffffffa05eca55&gt;] tgt_request_handle+0xaf5/0x1590 [ptlrpc] [48792.686829] [&lt;ffffffffa01eaf97&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [48792.686829] [&lt;ffffffffa0590eb6&gt;] ptlrpc_server_handle_request+0x256/0xad0 [ptlrpc] [48792.686829] [&lt;ffffffff810b9398&gt;] ? __wake_up_common+0x58/0x90 [48792.686829] [&lt;ffffffff813ccd2b&gt;] ? do_raw_spin_unlock+0x4b/0x90 [48792.686829] [&lt;ffffffffa0594cae&gt;] ptlrpc_main+0xabe/0x1f80 [ptlrpc] [48792.686829] [&lt;ffffffffa05941f0&gt;] ? ptlrpc_register_service+0xeb0/0xeb0 [ptlrpc] [48792.686829] [&lt;ffffffff810ae864&gt;] kthread+0xe4/0xf0 [48792.686829] [&lt;ffffffff810ae780&gt;] ? kthread_create_on_node+0x140/0x140 [48792.686829] [&lt;ffffffff81783777&gt;] ret_from_fork_nospec_begin+0x21/0x21 [48792.686829] [&lt;ffffffff810ae780&gt;] ? kthread_create_on_node+0x140/0x140 [48792.686829] Code: a0 31 c0 e8 53 be e9 ff 0f 1f 00 48 8b 03 be 01 00 00 00 48 8b 7d c0 48 8b 40 20 ff 50 18 e9 5a fe ff ff 0f 1f 84 00 00 00 00 00 &lt;49&gt; 8b 46 10 a8 01 0f 84 46 fe ff ff 48 8b 7d b0 31 c9 31 d2 be [48792.686829] RIP [&lt;ffffffffa034f110&gt;] lu_object_put+0x270/0x3c0 [obdclass] [48792.686829] RSP &lt;ffff8800b93afb38&gt; [48792.686829] CR2: ffff88008278be60 </pre> </div></div> LU-11204

mdt_reint_unlink->lu_object_put() crash

Bug Minor Resolved Fixed Mikhail Pershin Oleg Drokin Fri, 3 Aug 2018 16:45:55 +0000 Thu, 21 Nov 2019 21:40:20 +0000 Fri, 7 Jun 2019 12:09:29 +0000 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.12.4 0 5 <p>Can you use GDB to decode the line number and structure pointer to see exactly where it is crashing?</p> <p>The last change to that part of the code is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 478be95b8d938498ccf03920f934a0d49fe5dc6b Author: NeilBrown &lt;neilb@suse.com&gt; AuthorDate: Tue May 8 22:46:29 2018 -0400 LU-4423 obd: backport of lu_object changes upstream fold lu_object_new() into lu_object_find_at() lu_object_new() duplicates a lot of code that is in lu_object_find_at(). There is no real need for a separate function, it is simpler just to skip the bits of lu_object_find_at() that we don't want in the LOC_F_NEW case. Linux-commit: 775c4dc274343e5e2959fa1171baf2fc01028840 discard extra lru count. lu_object maintains 2 lru counts. One is a per-bucket lsb_lru_len. The other is the per-cpu ls_lru_len_counter. The only times the per-bucket counters are use are: - a debug message when an object is added - in lu_site_stats_get when all the counters are combined. The debug message is not essential, and the per-cpu counter can be used to get the combined total. So discard the per-bucket lsb_lru_len. Change-Id: I26203f331a0c73ae4e23878eb10b15d9fcf546c5 Signed-off-by: NeilBrown &lt;neilb@suse.com&gt; Signed-off-by: James Simmons &lt;uja.ornl@yahoo.com&gt; Reviewed-on: https://review.whamcloud.com/32325 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(gdb) l *(lu_object_put+0x275) 0x4f145 is in lu_object_put (/home/green/git/lustre-release/lustre/obdclass/lu_object.c:164). 159 160 cfs_hash_bd_get(site-&gt;ls_obj_hash, &amp;top-&gt;loh_fid, &amp;bd); 161 bkt = cfs_hash_bd_extra_get(site-&gt;ls_obj_hash, &amp;bd); 162 163 if (!cfs_hash_bd_dec_and_lock(site-&gt;ls_obj_hash, &amp;bd, &amp;top-&gt;loh_ref)) { 164 if (lu_object_is_dying(top)) { 165 /* </pre> </div></div> <p>(0x270 is some sort of a test_bit)</p> <p>so it's a bit hard to know where exctly did it crash I guess. Note we onl recently started to do real multimountpoint racer testing after a testscript fix from John so it's not necessary a super new regression, just possibly only recently exposed.</p> <p>ok, so it is line 163:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /home/green/bk/linux-3.10.0-862.3.2.el7-debug/./arch/x86/include/asm/bitops.h: 319 #11 [ffff8800b93afb30] lu_object_put at ffffffffa034efe8 [obdclass] /home/green/git/lustre-release/lustre/obdclass/lu_object.c: 163 #12 [ffff8800b93afb90] mdt_reint_unlink at ffffffffa0cbbb13 [mdt] /home/green/git/lustre-release/libcfs/include/libcfs/libcfs_debug.h: 146 #13 [ffff8800b93afc10] mdt_reint_rec at ffffffffa0cbfc10 [mdt] /home/green/git/lustre-release/lustre/mdt/mdt_reint.c: 2375 #14 [ffff8800b93afc38] mdt_reint_internal at ffffffffa0c9f6ab [mdt] /home/green/git/lustre-release/libcfs/include/libcfs/libcfs_debug.h: 146 </pre> </div></div> <p>but because it's a macro, it's a bit harder to know where exactly did it hit.</p> <p>it looks similar to <a href="https://jira.whamcloud.com/browse/LU-9942" title="Use after free in mdt_mfd_close-&gt;lu_object_put" class="issue-link" data-issue-key="LU-9942">LU-9942</a></p> <p>and more older <a href="https://jira.whamcloud.com/browse/LU-9419" title="mdt_intent_layout -&gt; lu_object_put crash" class="issue-link" data-issue-key="LU-9419">LU-9419</a></p> <p>each ticket has trace ending at <tt>lu_object_put</tt> line</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (!cfs_hash_bd_dec_and_lock(site-&gt;ls_obj_hash, &amp;bd, &amp;top-&gt;loh_ref)) { ---&gt; <span class="code-keyword">if</span> (lu_object_is_dying(top)) { </pre> </div></div> <p>This code path is about exit when loh_ref is not the last one, at the same time that looks like <tt>top</tt> is already destroyed at the moment of check</p> <p>The reason is the accessing <tt>top</tt> after atomic_dec_and_lock() call, at that moment <tt>top</tt> dropped own reference and is not protected so can be freed by other thread. Issue is being seen mostly on onyx-68 with many virtual machines running on the same node.<br/> Solution can be just getting lu_object_is_dying() value before loh_ref decrement, moreover I am not sure we need this whole block of code with </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (lu_object_is_dying(top)) { /* * somebody may be waiting <span class="code-keyword">for</span> <span class="code-keyword">this</span>, currently only * used <span class="code-keyword">for</span> cl_object, see cl_object_put_last(). */ wake_up_all(&amp;bkt-&gt;lsb_marche_funebre); } </pre> </div></div> <p>it is bz22520 <a href="https://bugzilla.lustre.org/show_bug.cgi?id=22520" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=22520</a> and it is worth to review how things are working now and if that wake_up() in lu_object_put() is needed for every put really.</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34960" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34960</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11204" title="mdt_reint_unlink-&gt;lu_object_put() crash" class="issue-link" data-issue-key="LU-11204"><del>LU-11204</del></a> obdclass: remove unprotected access to lu_object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 98c1b95c49b79509c7d31f2cdebdc46eda54a8b4</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34961" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34961</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11204" title="mdt_reint_unlink-&gt;lu_object_put() crash" class="issue-link" data-issue-key="LU-11204"><del>LU-11204</del></a> obdclass: remove unprotected access to lu_object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7992d3d38e148f0f9c60a750ba5355413e8b1407</p> <p>I've pushed two patches, first is simple to prevent after-free access by using local variable, second patch is fortestonly to check if cl_object_put_last() is still needed. At quick view conditions described in bz22520 don't exist in current code, so whole bz22520 fix might be not needed.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34960/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34960/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11204" title="mdt_reint_unlink-&gt;lu_object_put() crash" class="issue-link" data-issue-key="LU-11204"><del>LU-11204</del></a> obdclass: remove unprotected access to lu_object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 336cf0f2f3a9ce5b11a34aeaeec062a5d5144213</p> <p>So what's the verdict from <a href="https://review.whamcloud.com/#/c/34961/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34961/</a> ? Is further work needed or can this ticket be marked as RESOLVED?</p> <p>that was alternative approach, I've abandoned it. </p> <p>ok. Should we consider this fix for b2_12?</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36217" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36217</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11204" title="mdt_reint_unlink-&gt;lu_object_put() crash" class="issue-link" data-issue-key="LU-11204"><del>LU-11204</del></a> obdclass: remove unprotected access to lu_object<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7db34b95768d7f2df3aa110275ea26d345431852</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36217/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36217/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11204" title="mdt_reint_unlink-&gt;lu_object_put() crash" class="issue-link" data-issue-key="LU-11204"><del>LU-11204</del></a> obdclass: remove unprotected access to lu_object<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: e548e31f3feac2831868fe01cc75bf111cf8f501</p> Duplicate LU-12741 Related LU-9942 Development Rank 1|i0008v: Rank (Obsolete) 9223372036854775807