https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11122 Lustre <p>MDS nodes were power cycled during hardware maintenance. After they came back up, got below (some material redacted, see comments below for full console log contents):</p> <p>BUG: unable to handle kernel NULL pointer dereference at 0000000000000018<br/> IP: <span class="error">&#91;&lt;ffffffffc0cdc392&gt;&#93;</span> fld_local_lookup+0x52/0x270 <span class="error">&#91;fld&#93;</span><br/> CPU: 17 PID: 180501 Comm: orph_cleanup_ls Kdump: loaded Tainted: P OE ------------ 3.10.0-862.3.2.1chaos.ch6.x86_64 #1<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffc06e8f6c&gt;&#93;</span> ? dmu_tx_hold_object_impl+0x6c/0xc0 <span class="error">&#91;zfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc109ff28&gt;&#93;</span> osd_fld_lookup+0x48/0xd0 <span class="error">&#91;osd_zfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc10a008a&gt;&#93;</span> fid_is_on_ost+0xda/0x2f0 <span class="error">&#91;osd_zfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc10a02e9&gt;&#93;</span> osd_get_name_n_idx+0x49/0xd00 <span class="error">&#91;osd_zfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc109902c&gt;&#93;</span> ? osd_declare_attr_set+0x14c/0x730 <span class="error">&#91;osd_zfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc0753b7e&gt;&#93;</span> ? zap_lookup_by_dnode+0x2e/0x30 <span class="error">&#91;zfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc1097510&gt;&#93;</span> osd_declare_object_destroy+0xe0/0x3e0 <span class="error">&#91;osd_zfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc1139ffe&gt;&#93;</span> lod_sub_object_declare_destroy+0xce/0x2d0 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc1129700&gt;&#93;</span> lod_declare_object_destroy+0x170/0x4a0 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc1513689&gt;&#93;</span> ? orph_declare_index_delete+0x179/0x460 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc1513f66&gt;&#93;</span> orph_key_test_and_del+0x5f6/0xd30 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc1514c57&gt;&#93;</span> __mdd_orphan_cleanup+0x5b7/0x840 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc15146a0&gt;&#93;</span> ? orph_key_test_and_del+0xd30/0xd30 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffbb2c05f1&gt;&#93;</span> kthread+0xd1/0xe0<br/> <span class="error">&#91;&lt;ffffffffbb2c0520&gt;&#93;</span> ? insert_kthread_work+0x40/0x40<br/> <span class="error">&#91;&lt;ffffffffbb9438b7&gt;&#93;</span> ret_from_fork_nospec_begin+0x21/0x21<br/> <span class="error">&#91;&lt;ffffffffbb2c0520&gt;&#93;</span> ? insert_kthread_work+0x40/0x40</p> lustre-2.8.2_2.chaos-1.ch6.x86_64 <br/> 4 MDTs used in DNE-1 fashion (remote dirs, no striped dirs) <br/> RHEL 7.5 LU-11122

NULL pointer dereference in fld_local_lookup()

Bug Blocker Closed Duplicate Alex Zhuravlev Olaf Faaland llnl Thu, 5 Jul 2018 19:46:46 +0000 Wed, 8 Aug 2018 17:06:11 +0000 Wed, 8 Aug 2018 17:05:56 +0000 0 5 <p>This a production file system, it's currently down due to this issue.</p> <p>More complete console log contents:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[2507417.597913] Lustre: lsrza-MDT0003: Recovery over after 1:45, of 1011 clients 0 recovered and 1011 were evicted. [2507417.634717] LustreError: 162832:0:(client.c:1164:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9eaed88fef00 x1602545796916964/t0(0) o5-&gt;lsrza-OST0003-osc-MDT0003@172.21.3.8@o2ib700:28/4 lens 432/432 e 0 to 0 dl 0 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 [2507417.634723] LustreError: 162879:0:(osp_precreate.c:900:osp_precreate_cleanup_orphans()) lsrza-OST0005-osc-MDT0003: cannot cleanup orphans: rc = -5 [2507417.634725] LustreError: 162963:0:(osp_precreate.c:900:osp_precreate_cleanup_orphans()) lsrza-OST0007-osc-MDT0003: cannot cleanup orphans: rc = -5 [2507417.634726] LustreError: 162879:0:(osp_precreate.c:900:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [2507417.634727] LustreError: 162963:0:(osp_precreate.c:900:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [2507417.724392] LustreError: 162832:0:(client.c:1164:ptlrpc_import_delay_req()) Skipped 926 previous similar messages [2507417.726598] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 [2507417.726603] IP: [&lt;ffffffffc0cdc392&gt;] fld_local_lookup+0x52/0x270 [fld] [2507417.726604] PGD 0 [2507417.726605] Oops: 0000 [#1] SMP &lt;&lt;Long line listing Modules linked in redacted&gt;&gt; [2507417.726650] CPU: 17 PID: 180501 Comm: orph_cleanup_ls Kdump: loaded Tainted: P OE ------------ 3.10.0-862.3.2.1chaos.ch6.x86_64 #1 [2507417.726651] Hardware name: Intel Corporation S2600WTTR/S2600WTTR, BIOS SE5C610.86B.01.01.0019.101220160604 10/12/2016 [2507417.726651] task: ffff9eacbe858000 ti: ffff9ea134cdc000 task.ti: ffff9ea134cdc000 [2507417.726655] RIP: 0010:[&lt;ffffffffc0cdc392&gt;] [&lt;ffffffffc0cdc392&gt;] fld_local_lookup+0x52/0x270 [fld] [2507417.726655] RSP: 0018:ffff9ea134cdfc00 EFLAGS: 00010286 [2507417.726656] RAX: ffff9e7059e223c0 RBX: ffff9e7059e223c0 RCX: ffff9eaff5e9d430 [2507417.726657] RDX: ffff9e6e32fa4800 RSI: ffffffffc0ce4ec0 RDI: ffff9eb0007e9240 [2507417.726657] RBP: ffff9ea134cdfc38 R08: 00000002c000f1db R09: ffff9e32bfc07a00 [2507417.726658] R10: ffff9e32bfc07a00 R11: ffffffffc03d4208 R12: ffff9eaff5e9d430 [2507417.726658] R13: 0000000000000000 R14: 00000002c000f1e7 R15: ffff9eaf7318c000 [2507417.726659] FS: 0000000000000000(0000) GS:ffff9e70be840000(0000) knlGS:0000000000000000 [2507417.726660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [2507417.726660] CR2: 0000000000000018 CR3: 0000003f7e378000 CR4: 00000000001607e0 [2507417.726661] Call Trace: [2507417.726689] [&lt;ffffffffc06e8f6c&gt;] ? dmu_tx_hold_object_impl+0x6c/0xc0 [zfs] [2507417.726696] [&lt;ffffffffc109ff28&gt;] osd_fld_lookup+0x48/0xd0 [osd_zfs] [2507417.726700] [&lt;ffffffffc10a008a&gt;] fid_is_on_ost+0xda/0x2f0 [osd_zfs] [2507417.726703] [&lt;ffffffffc10a02e9&gt;] osd_get_name_n_idx+0x49/0xd00 [osd_zfs] [2507417.726706] [&lt;ffffffffc109902c&gt;] ? osd_declare_attr_set+0x14c/0x730 [osd_zfs] [2507417.726729] [&lt;ffffffffc0753b7e&gt;] ? zap_lookup_by_dnode+0x2e/0x30 [zfs] [2507417.726733] [&lt;ffffffffc1097510&gt;] osd_declare_object_destroy+0xe0/0x3e0 [osd_zfs] [2507417.726742] [&lt;ffffffffc1139ffe&gt;] lod_sub_object_declare_destroy+0xce/0x2d0 [lod] [2507417.726748] [&lt;ffffffffc1129700&gt;] lod_declare_object_destroy+0x170/0x4a0 [lod] [2507417.726756] [&lt;ffffffffc1513689&gt;] ? orph_declare_index_delete+0x179/0x460 [mdd] [2507417.726760] [&lt;ffffffffc1513f66&gt;] orph_key_test_and_del+0x5f6/0xd30 [mdd] [2507417.726776] [&lt;ffffffffc1514c57&gt;] __mdd_orphan_cleanup+0x5b7/0x840 [mdd] [2507417.726780] [&lt;ffffffffc15146a0&gt;] ? orph_key_test_and_del+0xd30/0xd30 [mdd] [2507417.726782] [&lt;ffffffffbb2c05f1&gt;] kthread+0xd1/0xe0 [2507417.726783] [&lt;ffffffffbb2c0520&gt;] ? insert_kthread_work+0x40/0x40 [2507417.726786] [&lt;ffffffffbb9438b7&gt;] ret_from_fork_nospec_begin+0x21/0x21 [2507417.726786] [&lt;ffffffffbb2c0520&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>Top of stack:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[2507417.726696] [&lt;ffffffffc109ff28&gt;] osd_fld_lookup+0x48/0xd0 [osd_zfs] [2507417.726700] [&lt;ffffffffc10a008a&gt;] fid_is_on_ost+0xda/0x2f0 [osd_zfs] [2507417.726703] [&lt;ffffffffc10a02e9&gt;] osd_get_name_n_idx+0x49/0xd00 [osd_zfs] </pre> </div></div> <p>And point of NULL ptr dereference:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[2507417.726655] RIP: 0010:[&lt;ffffffffc0cdc392&gt;] [&lt;ffffffffc0cdc392&gt;] fld_local_lookup+0x52/0x270 [fld] </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(gdb) l *(fld_local_lookup+0x52) 0x53c2 is in fld_local_lookup (/usr/src/debug/lustre-2.8.2_2.chaos/lustre/fld/fld_handler.c:218). 213 info = lu_context_key_get(&amp;env-&gt;le_ctx, &amp;fld_thread_key); 214 LASSERT(info != NULL); 215 erange = &amp;info-&gt;fti_lrange; 216 217 /* Lookup it in the cache. */ 218 rc = fld_cache_lookup(fld-&gt;lsf_cache, seq, erange); 219 if (rc == 0) { 220 if (unlikely(fld_range_type(erange) != fld_range_type(range) &amp;&amp; 221 !fld_range_is_any(range))) { 222 CERROR("%s: FLD cache range "DRANGE" does not match" </pre> </div></div> <p>Variable info cannot be NULL, because it is local, not shared, and the ASSERT verified it was not NULL after it was last assigned. So it seems like line 218 really is where the NULL ptr deref is occurring:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>218 rc = fld_cache_lookup(fld-&gt;lsf_cache, seq, erange); </pre> </div></div> <p>fld was passed in and we have not check it for NULL, and that's the only pointer used. So it seems fld must be NULL.</p> <p> </p> <p>For our code, including the tags we build from, see the lustre-release-fe-llnl project in gerrit.</p> <p> </p> <p> </p> <p>Is there a way we can tell the MDT to skip orphan cleanup once, so it forgets about these orphaned objects?  We don't really care if some <b>unused</b> objects are orphaned.</p> <p>&gt; Is there a way we can tell the MDT to skip orphan cleanup once, so it forgets about these orphaned objects? We don't really care if some unused objects are orphaned.</p> <p>You could mount the backing FS and remove the files from the PENDING directory.</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ofaaland" class="user-hover" rel="ofaaland">ofaaland</a>,</p> <p>Is this a Lustre only system or do you also have IML installed?</p> <p>Regards,</p> <p>Will</p> <p>Lustre only, no IML.</p> <p>The file system is back up.  This is no longer an emergency.</p> <p>In the course of rebooting the nodes and mounting manually to get more information, the messages about orphan cleanup failing stopped appearing after recovery.  It's been about 6 minutes now since the MDTs completed recovery.</p> <p>John, thanks for the suggestion.  I'll try it if this comes up again.</p> <p>Thank you for responding so quickly.</p> <p>No problem. We'll discuss tomorrow whether further work could be useful to be able to avoid this kind of scenario.</p> <p>Alex</p> <p>What do you recommend here?</p> <p>Peter</p> <p>was it possible that umount was initiated at that time? I see that 2.8.2_2.chaos stops orphan thread in mdd_shutdown() (LCFG_CLEANUP path) while master branch stops orphan thread in the preceding phase (LCFG_PRE_CLEANUP path)</p> <p> </p> <p>in  2.8.2_2.chaos the sequence at umount will be:</p> <p>mdt_fini() <del>&gt; mdt_fld_fini() ( ss</del>&gt;ss_server_fld = NULL ) -&gt; mdt_stack_fini() -&gt; ldo_process_config*LCFG_CLEANUP) -&gt; mdd_device_shutdown() -&gt; mdd_generic_thread_stop(mdd_orph_cleanup_thread)</p> <p>IOW, ss_server_fld can be reset to NULL while the orphan thread is still running</p> <p> </p> <p>please check <a href="http://review.whamcloud.com/23029" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23029</a> </p> <p>You're right, this was in the context of a umount.  That all seems to fit.</p> <p> </p> <p>Is any further action needed here or can this ticket be marked as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7206" title="LBUG osp_sync.c:1541:osp_sync_id_traction_fini()) ASSERTION( list_empty(&amp;tr-&gt;otr_wakeup_list) ) failed: " class="issue-link" data-issue-key="LU-7206"><del>LU-7206</del></a>?</p> <p>Peter, can you backport this to 2.8fe?  This way it gets some automated sanity tests run against it, which we aren't really set up to do.</p> <p>Backport reviewed and passed testing.</p> <p>Dupe of <a href="https://jira.whamcloud.com/browse/LU-7206" title="LBUG osp_sync.c:1541:osp_sync_id_traction_fini()) ASSERTION( list_empty(&amp;tr-&gt;otr_wakeup_list) ) failed: " class="issue-link" data-issue-key="LU-7206"><del>LU-7206</del></a></p> <p>Dupe of <a href="https://jira.whamcloud.com/browse/LU-7206" title="LBUG osp_sync.c:1541:osp_sync_id_traction_fini()) ASSERTION( list_empty(&amp;tr-&gt;otr_wakeup_list) ) failed: " class="issue-link" data-issue-key="LU-7206"><del>LU-7206</del></a></p> Development Rank 1|hzzysv: Rank (Obsolete) 9223372036854775807 Severity