https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13018 Lustre <p>We monitor changelog_size on our MDTs, fetching it every 5 minutes under normal operation.  We also consume changelogs constantly via Starfish.  We occasionally see the following BUG:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: 29422:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages BUG: unable to handle kernel NULL pointer dereference at 000000000000001c IP: [&lt;ffffffffc14d20a2&gt;] osd_attr_get+0x62/0x340 [osd_zfs] PGD 0 Oops: 0000 [#1] SMP CPU: 4 PID: 21384 Comm: snmpd Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1062.1.1.1chaos.ch6.x86_64 #1 Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016 task: ffff9b1a2ab2b150 ti: ffff9b170ece0000 task.ti: ffff9b170ece0000 RIP: 0010:[&lt;ffffffffc14d20a2&gt;]  [&lt;ffffffffc14d20a2&gt;] osd_attr_get+0x62/0x340 [osd_zfs] Call Trace: [&lt;ffffffffc12f73b5&gt;] llog_size+0x35/0xe0 [obdclass] [&lt;ffffffffc13006fe&gt;] ? llog_cat_id2handle+0x30e/0x5b0 [obdclass] [&lt;ffffffffb9232fb8&gt;] ? kmem_cache_alloc+0x48/0x240 [&lt;ffffffffc1300b9c&gt;] llog_cat_size_cb+0x1fc/0x3d0 [obdclass] [&lt;ffffffffc12fb9db&gt;] llog_process_thread+0x87b/0x1470 [obdclass] [&lt;ffffffffb9232c89&gt;] ? ___slab_alloc+0x209/0x4f0 [&lt;ffffffffc13009a0&gt;] ? llog_cat_id2handle+0x5b0/0x5b0 [obdclass] [&lt;ffffffffc12fc68c&gt;] llog_process_or_fork+0xbc/0x450 [obdclass] [&lt;ffffffffc13009a0&gt;] ? llog_cat_id2handle+0x5b0/0x5b0 [obdclass] [&lt;ffffffffc1300f09&gt;] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [&lt;ffffffffc1301098&gt;] llog_cat_size+0x58/0x80 [obdclass] [&lt;ffffffffc19d7a32&gt;] mdd_changelog_size_ctxt+0x92/0x320 [mdd] [&lt;ffffffffc19d7d35&gt;] mdd_changelog_size_seq_show+0x75/0xe0 [mdd] [&lt;ffffffffb9284d18&gt;] seq_read+0x138/0x460 [&lt;ffffffffb92d94b0&gt;] proc_reg_read+0x40/0x80 [&lt;ffffffffb925a71c&gt;] vfs_read+0xbc/0x1c0 [&lt;ffffffffb925b68f&gt;] SyS_read+0x7f/0xf0 [&lt;ffffffffb97ba11e&gt;] system_call_fastpath+0x25/0x2a </pre> </div></div> zfs-0.7.11-9llnl <br/> Lustre 2.10.8_4.chaos <br/> See <a href="https://github.com/LLNL/lustre/commits/2.10.8_4.chaos">https://github.com/LLNL/lustre/commits/2.10.8_4.chaos</a> <br/> RHEL 7.7 based OS LU-13018

NULL pointer dereference in osd_attr_get while consuming changelogs and reading changelog_size

Task Minor Resolved Duplicate Mikhail Pershin Olaf Faaland llnl Tue, 26 Nov 2019 18:25:59 +0000 Sat, 1 Feb 2020 16:13:21 +0000 Sat, 1 Feb 2020 16:13:13 +0000 Lustre 2.10.8 0 3 <p>I haven't tried the obvious thing of reading changelog_size in a tight loop while clearing changelogs to see how easy it is to reproduce.  I'll do that today.</p> <p>Mike</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>I wasn't able to quickly reproduce it as described above.  I'll let you know if running for a longer duration works.</p> <p>Olaf, is that possible to find out what is happening at 0x62 there:<br/> <tt>RIP: 0010:<span class="error">&#91;&lt;ffffffffc14d20a2&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffc14d20a2&gt;&#93;</span> osd_attr_get+0x62/0x340 <span class="error">&#91;osd_zfs&#93;</span></tt><br/> or maybe attach osd_object.o binary file or .ko module?</p> <p>Mike,</p> <p>Sorry, I meant to do that in the first place.  Here you go:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(gdb) l *(osd_attr_get+0x62) 0x90d2 is in osd_attr_get (/usr/src/debug/lustre-2.10.8_4.chaos/lustre/osd-zfs/osd_object.c:788). 783 uint32_t blksize; 784 int rc = 0; 785 786 down_read(&amp;obj-&gt;oo_guard); 787 788 if (unlikely(!dt_object_exists(dt) || obj-&gt;oo_destroyed)) 789 GOTO(out, rc = -ENOENT); 790 791 if (unlikely(fid_is_acct(lu_object_fid(&amp;dt-&gt;do_lu)))) 792 GOTO(out, rc = 0); </pre> </div></div> <p>Thanks</p> <p>As I understand this offset 0x1c is for <tt>loh_attr</tt> field in lo_header while checking <tt>dt_object_exists()</tt>. That means we have a race here between llog deletion and polling its size. I will check related code.</p> <p>Olaf, can this be reproduced easily or this is quite rare situation?</p> <p>Hi Mike,</p> <p>Thanks for looking.  I haven't been able to reproduce it so far.  It has not been frequent in production, but has happened multiple times.</p> <p> </p> <p>Olaf, speaking of possible use-after-free for <tt>lu_object</tt>, there is NULL pointer and I wonder why it is not poisoned, can you check kernel config options for SLAB_DEBUG or any other poisoning things? Are they turned off? I mean that if allocation debug is being used then this case is unlikely use-after-free</p> <p>Hi Mike,</p> <p>I know of the concept but don't know the different kernel config options to search for.  Here's one - let me know what others to look for.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@zinc2:~]# grep SLUB_DEBUG /boot/config-3.10.0-1062.1.1.1chaos.ch6.x86_64 CONFIG_SLUB_DEBUG=y # CONFIG_SLUB_DEBUG_ON is not set</pre> </div></div> <p>I confirmed that all of /sys/kernel/slab/*/poison are "0", which I believe means no poisoning is being done for that memory.  If I'm wrong, let me know.</p> <p>The kernel is a patched version of RHEL 7.7 kernel, 3.10.0-1062.1.1.1chaos.ch6.x86_64.</p> <p> </p> <p>For my reference: My local issue is TOSS4656</p> <p>this issue has the same reason as <a href="https://jira.whamcloud.com/browse/LU-10198" title="GPF llog_osd_declare_write_rec+0xb6/0x3d0 " class="issue-link" data-issue-key="LU-10198"><del>LU-10198</del></a>, I will prepate patch soon</p> <p>Patch was added under <a href="https://jira.whamcloud.com/browse/LU-10198" title="GPF llog_osd_declare_write_rec+0xb6/0x3d0 " class="issue-link" data-issue-key="LU-10198"><del>LU-10198</del></a>, here is the link: <a href="https://review.whamcloud.com/#/c/37367/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/37367/</a></p> <p>Thanks, Mikhail.  You can close this issue as dupe.</p> Related LU-10198 Development Rank 1|i00pzz: Rank (Obsolete) 9223372036854775807