https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3917 Lustre <p>MDS crashes with summary assertion after upgrade from 2.1 to 2.4. Added -o abort_recov mount option but MDS still crashes each time it is started.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 4684 TASK: ffff880628b33540 CPU: 12 COMMAND: "tgt_recov" #0 [ffff880628b35a48] machine_kexec at ffffffff81035fcb #1 [ffff880628b35aa8] crash_kexec at ffffffff810c10b2 #2 [ffff880628b35b78] panic at ffffffff81510333 #3 [ffff880628b35bf8] lbug_with_loc at ffffffffa0507f4b [libcfs] #4 [ffff880628b35c18] lod_initialize_objects at ffffffffa1141d3b [lod] #5 [ffff880628b35ca8] lod_parse_striping at ffffffffa11421e1 [lod] #6 [ffff880628b35cd8] lod_load_striping at ffffffffa1143c44 [lod] #7 [ffff880628b35d18] lod_declare_object_destroy at ffffffffa114f6db [lod] #8 [ffff880628b35d48] __mdd_orphan_cleanup at ffffffffa0e190a9 [mdd] #9 [ffff880628b35de8] mdd_recovery_complete at ffffffffa0e2833d [mdd] #10 [ffff880628b35e18] mdt_postrecov at ffffffffa1079cb5 [mdt] #11 [ffff880628b35e38] mdt_obd_postrecov at ffffffffa107b178 [mdt] #12 [ffff880628b35ea8] target_recovery_thread at ffffffffa09a6ca4 [ptlrpc] #13 [ffff880628b35f48] kernel_thread at ffffffff8100c10a </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ZFS: Loaded module v0.6.2-1.2, ZFS pool version 5000, ZFS filesystem version 5 Lustre: Lustre: Build Version: 2.4.0-15chaos-15chaos--PRISTINE-2.6.32-358.14.1.2chaos.ch5.1.1.x86_64 LDISKFS-fs (sdb): mounted filesystem with ordered data mode. quota=off. Opts: Lustre: lsc-MDT0000: Not available for connect from 192.168.117.178@o2ib10 (not set up) Lustre: 4673:0:(mdt_handler.c:4947:mdt_process_config()) For interoperability, skip this mdd.quota_type. It is obsolete. LustreError: 11-0: lsc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. LustreError: 4469:0:(mdt_handler.c:5930:mdt_iocontrol()) lsc-MDT0000: Aborting recovery for device Lustre: lsc-MDT0000: Aborting recovery LustreError: 4684:0:(lod_lov.c:706:lod_initialize_objects()) ASSERTION( cfs_bitmap_check(md-&gt;lod_ost_descs.ltd_tgt_bitmap, idx) ) failed: LustreError: 4684:0:(lod_lov.c:706:lod_initialize_objects()) LBUG Pid: 4684, comm: tgt_recov Call Trace: [&lt;ffffffffa05078f5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0507ef7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa1141d3b&gt;] lod_initialize_objects+0x98b/0xc30 [lod] [&lt;ffffffffa11421e1&gt;] lod_parse_striping+0x201/0x300 [lod] [&lt;ffffffffa1143c44&gt;] lod_load_striping+0x2a4/0x4b0 [lod] [&lt;ffffffffa114f6db&gt;] lod_declare_object_destroy+0x16b/0x390 [lod] [&lt;ffffffffa0e190a9&gt;] __mdd_orphan_cleanup+0x7d9/0xca0 [mdd] [&lt;ffffffffa0e2833d&gt;] mdd_recovery_complete+0xed/0x170 [mdd] [&lt;ffffffffa1079cb5&gt;] mdt_postrecov+0x35/0xd0 [mdt] [&lt;ffffffffa107b178&gt;] mdt_obd_postrecov+0x78/0x90 [mdt] [&lt;ffffffffa09964e0&gt;] ? ldlm_reprocess_res+0x0/0x20 [ptlrpc] [&lt;ffffffffa099189e&gt;] ? ldlm_reprocess_all_ns+0x3e/0x110 [ptlrpc] [&lt;ffffffffa09a6ca4&gt;] target_recovery_thread+0xc64/0x1980 [ptlrpc] [&lt;ffffffffa09a6040&gt;] ? target_recovery_thread+0x0/0x1980 [ptlrpc] [&lt;ffffffff8100c10a&gt;] child_rip+0xa/0x20 [&lt;ffffffffa09a6040&gt;] ? target_recovery_thread+0x0/0x1980 [ptlrpc] [&lt;ffffffffa09a6040&gt;] ? target_recovery_thread+0x0/0x1980 [ptlrpc] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG Pid: 4684, comm: tgt_recov Tainted: P --------------- 2.6.32-358.14.1.2chaos.ch5.1.1.x86_64 #1 Call Trace: [&lt;ffffffff8151032c&gt;] ? panic+0xa7/0x16f [&lt;ffffffffa0507f4b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa1141d3b&gt;] ? lod_initialize_objects+0x98b/0xc30 [lod] [&lt;ffffffffa11421e1&gt;] ? lod_parse_striping+0x201/0x300 [lod] [&lt;ffffffffa1143c44&gt;] ? lod_load_striping+0x2a4/0x4b0 [lod] [&lt;ffffffffa114f6db&gt;] ? lod_declare_object_destroy+0x16b/0x390 [lod] [&lt;ffffffffa0e190a9&gt;] ? __mdd_orphan_cleanup+0x7d9/0xca0 [mdd] [&lt;ffffffffa0e2833d&gt;] ? mdd_recovery_complete+0xed/0x170 [mdd] [&lt;ffffffffa1079cb5&gt;] ? mdt_postrecov+0x35/0xd0 [mdt] [&lt;ffffffffa107b178&gt;] ? mdt_obd_postrecov+0x78/0x90 [mdt] [&lt;ffffffffa09964e0&gt;] ? ldlm_reprocess_res+0x0/0x20 [ptlrpc] [&lt;ffffffffa099189e&gt;] ? ldlm_reprocess_all_ns+0x3e/0x110 [ptlrpc] [&lt;ffffffffa09a6ca4&gt;] ? target_recovery_thread+0xc64/0x1980 [ptlrpc] [&lt;ffffffffa09a6040&gt;] ? target_recovery_thread+0x0/0x1980 [ptlrpc] [&lt;ffffffff8100c10a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffffa09a6040&gt;] ? target_recovery_thread+0x0/0x1980 [ptlrpc] [&lt;ffffffffa09a6040&gt;] ? target_recovery_thread+0x0/0x1980 [ptlrpc] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 REWRITING MCP55 CFG REG CFG = c1 </pre> </div></div> lustre-2.4.0-15chaos_2.6.32_358.14.1.2chaos.ch5.1.1.ch5.1.1.x86_64 LU-3917

lod_initialize_objects()) ASSERTION( cfs_bitmap_check(md->lod_ost_descs.ltd_tgt_bitmap, idx) ) failed

Bug Blocker Resolved Duplicate Zhenyu Xu Ned Bass Tue, 10 Sep 2013 18:17:25 +0000 Wed, 16 Oct 2013 03:07:56 +0000 Mon, 23 Sep 2013 17:43:50 +0000 Lustre 2.4.0 Lustre 2.5.0 0 7 <p>Possibly related to <a href="https://jira.whamcloud.com/browse/LU-3161" title="LASSERT() in lod_initialize_objects()" class="issue-link" data-issue-key="LU-3161"><del>LU-3161</del></a></p> <p>Note that this issue is Severity 1. This filesystem is completely out of action until the problem is resolved.</p> <p>Oleg</p> <p>Can you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>This is indeed the same assert as referenced by <a href="https://jira.whamcloud.com/browse/LU-3161" title="LASSERT() in lod_initialize_objects()" class="issue-link" data-issue-key="LU-3161"><del>LU-3161</del></a> and the patch referenced there: <a href="http://review.whamcloud.com/#/c/7234/6" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7234/6</a> should stop this assertion, the added benefit is it would print the values the assertion is unhappy about.</p> <p>Also I assume you have a crashdump, can we get a debug log out of it (with some higher debug setting enabled)</p> <p>Another point that may be significant, it that this is an upgrade of a older ldiskfs filesystem that was probably formatted under 1.8 or earlier. As such, in order for the upgraded 2.4 filesystem to work with 2.4 clients, we needed to writeconf the filesystem.</p> <p>Perhaps the lack of any registered OSTs at this stage explains the assertion.</p> <p>I'm in the process of building a test tag with the <a href="https://jira.whamcloud.com/browse/LU-3161" title="LASSERT() in lod_initialize_objects()" class="issue-link" data-issue-key="LU-3161"><del>LU-3161</del></a> patch.</p> <p>FYI, Ned found that moving the PENDING directory out of the way allowed us to boot without a crash. We are going to put the patch in, restore the PENDING directory, and see if the patch allows a clean boot with the PENDING contents in place.</p> <p>Attached full debug log <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/13453/13453_lustre.log.LU-3917.gz" title="lustre.log.LU-3917.gz attached to LU-3917">lustre.log.LU-3917.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>The <a href="https://jira.whamcloud.com/browse/LU-3161" title="LASSERT() in lod_initialize_objects()" class="issue-link" data-issue-key="LU-3161"><del>LU-3161</del></a> patch appears to address this assertion. However we still don't make it past <tt>lod_initialize_ojbects()</tt> due to a NULL pointer deref reported in <a href="https://jira.whamcloud.com/browse/LU-3918" title="kernel NULL pointer dereference at lod_initialize_objects+0x327" class="issue-link" data-issue-key="LU-3918"><del>LU-3918</del></a>.</p> <p>So from the logs it looks like no attempts to bring up any osps were made, possibly due to old config log that don't list any targets?</p> <p>Can you please show what's in config logs? use llog_read to read it off directly mounted mgs filesystem.</p> <p>Hi Oleg, as Chris mentioned above we had performed a writeconf to clear the config logs.</p> <p>Ah, well, then that's why it crashed.</p> <p>anyway, crashing is not a normal behavior of course, so I'll update the fixing patch.<br/> Now the other problem is the object leakage is bound to occur here because the orphan cleanup would not happen (no OSTs connected to send destroy requests to).<br/> I wonder how to better address this.</p> <p>BTW, MGS is still supposed to regenerate the config logs, but I imagine you did not start OSTs before starting MDT?</p> <p>Oleg, that's right, we started the MDT/MGS before the OSTs.</p> <p>Our plan for future updates is to restart the MDS before writeconf to force orphan cleanup.</p> <p>Just to be sure I just reproduced this on master with these steps:<br/> 1. mount lustre<br/> 2. cat /dev/zero &gt;/mnt/lustre/file<br/> 3. ^Z<br/> 4. rm /mnt/lustre/file<br/> 5. umount /mnt/mds1<br/> 6. tunefs.lustre --writeconf /path/to/mds/device<br/> 7 mount /path/to/mds/device /mnt -t lustre</p> <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-3161" title="LASSERT() in lod_initialize_objects()" class="issue-link" data-issue-key="LU-3161"><del>LU-3161</del></a></p> <p>This is related to <a href="https://jira.whamcloud.com/browse/LU-3161" title="LASSERT() in lod_initialize_objects()" class="issue-link" data-issue-key="LU-3161"><del>LU-3161</del></a>, but not a duplicate. The <a href="https://jira.whamcloud.com/browse/LU-3161" title="LASSERT() in lod_initialize_objects()" class="issue-link" data-issue-key="LU-3161"><del>LU-3161</del></a> work does not go far enough to deal with the problem in this ticket.</p> <p>In this ticket, we need to deal with files in the PENDING directory after a writeconf of the filesystem (a writeconf is currently <em>required</em> when upgrading to 2.4, otherwise 2.4 clients won't work).</p> <p>FYI, we are not doing any further 2.4 server upgrades at LLNL until a couple of things are fixed, and this is one of the things that needs to be fixed.</p> <p>Bobijam</p> <p>Could you please continue with this effort - Oleg will be at LAD next week</p> <p>Thanks</p> <p>Peter</p> <p>I tried patch <a href="http://review.whamcloud.com/7234" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7234</a>, it addresses this issue, and I can't reproduce the issue stated in <a href="https://jira.whamcloud.com/browse/LU-3918" title="kernel NULL pointer dereference at lod_initialize_objects+0x327" class="issue-link" data-issue-key="LU-3918"><del>LU-3918</del></a>.</p> <p>Closing this as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-3161" title="LASSERT() in lod_initialize_objects()" class="issue-link" data-issue-key="LU-3161"><del>LU-3161</del></a>, which has a patch already.</p> Duplicate LU-4005 Related LU-3161 LU-3918 Development Rank 1|hzw1nr: Rank (Obsolete) 10350 Severity