https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4829 Lustre <p>We have our TDS system setup in wide-stripe mode. Each OSS is mounting over 100 OSTs. On mount the other day, we hit an assertion when scrub started.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[12319.230157] LustreError: 54554:0:(osd_internal.h:752:osd_fid2oi()) ASSERTION( !fid_is_idif(fid) ) failed: [0x100000000:0x1:0x0] [12319.242502] LustreError: 54554:0:(osd_internal.h:752:osd_fid2oi()) LBUG [12319.249538] Pid: 54554, comm: OI_scrub [12319.253707] [12319.253707] Call Trace: [12319.258529] [&lt;ffffffffa03dd895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [12319.265837] [&lt;ffffffffa03dde97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [12319.272395] [&lt;ffffffffa0be40f5&gt;] __osd_oi_lookup+0x3a5/0x3b0 [osd_ldiskfs] [12319.279770] [&lt;ffffffff8119dfcd&gt;] ? generic_drop_inode+0x1d/0x80 [12319.286133] [&lt;ffffffffa0be4174&gt;] osd_oi_lookup+0x74/0x140 [osd_ldiskfs] [12319.293197] [&lt;ffffffffa0bf8fbf&gt;] osd_scrub_exec+0x1af/0xf30 [osd_ldiskfs] [12319.300553] [&lt;ffffffffa0bfa5f2&gt;] ? osd_scrub_next+0x142/0x4b0 [osd_ldiskfs] [12319.308061] [&lt;ffffffffa0b71432&gt;] ? ldiskfs_read_inode_bitmap+0x172/0x2c0 [ldiskfs] [12319.316454] [&lt;ffffffffa0bf4d4f&gt;] osd_inode_iteration+0x1cf/0x570 [osd_ldiskfs] [12319.324461] [&lt;ffffffff810516b9&gt;] ? __wake_up_common+0x59/0x90 [12319.330764] [&lt;ffffffffa0bf8e10&gt;] ? osd_scrub_exec+0x0/0xf30 [osd_ldiskfs] [12319.337941] [&lt;ffffffffa0bfa4b0&gt;] ? osd_scrub_next+0x0/0x4b0 [osd_ldiskfs] [12319.345300] [&lt;ffffffffa0bf732a&gt;] osd_scrub_main+0x59a/0xd00 [osd_ldiskfs] [12319.352591] [&lt;ffffffff810097cc&gt;] ? __switch_to+0x1ac/0x320 [12319.358585] [&lt;ffffffffa0bf6d90&gt;] ? osd_scrub_main+0x0/0xd00 [osd_ldiskfs] [12319.365881] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [12319.371174] [&lt;ffffffffa0bf6d90&gt;] ? osd_scrub_main+0x0/0xd00 [osd_ldiskfs] [12319.378509] [&lt;ffffffffa0bf6d90&gt;] ? osd_scrub_main+0x0/0xd00 [osd_ldiskfs] [12319.385799] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>We had panic_on_lbug off, so we don't have a crash dump. But the system is still running, so if there's anything useful we can try to grab it. I tried to cat /proc/fs/lustre/osd-ldiskfs/atlastds-OST00f3/oi_scrub but it just hangs. That 'cat' process is stuck on the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat /proc/83715/stack [&lt;ffffffff81281f34&gt;] call_rwsem_down_read_failed+0x14/0x30 [&lt;ffffffffa0bf630d&gt;] osd_scrub_dump+0x3d/0x320 [osd_ldiskfs] [&lt;ffffffffa0be6055&gt;] lprocfs_osd_rd_oi_scrub+0x75/0xb0 [osd_ldiskfs] [&lt;ffffffffa054f563&gt;] lprocfs_fops_read+0xf3/0x1f0 [obdclass] [&lt;ffffffff811e9fee&gt;] proc_reg_read+0x7e/0xc0 [&lt;ffffffff81181f05&gt;] vfs_read+0xb5/0x1a0 [&lt;ffffffff81182041&gt;] sys_read+0x51/0x90 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>the FID it's complaining about <span class="error">&#91;0x100000000:0x1:0x0&#93;</span> looks suspect. The sequence is FID_SEQ_IDIF and the ObjID is 1. I know on ext4 inode 1 stores the bad blocks information, but I don't think that's what we're seeing here.</p> <p>We haven't yet tried to re-mount to see if the issue is persistent, since there may be something on the running system that you want us to provide. But we can do that if it's helpful.</p> LU-4829

LBUG: ASSERTION( !fid_is_idif(fid) )

Bug Minor Resolved Fixed nasf Matt Ezell mn4 Fri, 28 Mar 2014 14:38:58 +0000 Sun, 10 Aug 2014 12:32:07 +0000 Thu, 7 Aug 2014 14:22:15 +0000 Lustre 2.4.3 Lustre 2.5.0 0 7 <p>Is this a 2.4 formatted filesystem or was it created in the past with some other version and then upgraded?</p> <p>Sorry, I should have included that in the original report. It was formatted with 2.4, so there shouldn't be any IGIF/IDIF files.</p> <p>Fan Yong</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>With enabling LMA on OST-object for lustre-2.4.3, we need to back port the patch <a href="http://review.whamcloud.com/#/c/6669/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6669/</a></p> <p>Matt</p> <p>This fix is included in all 2.5.x releases. It would be possible to back port it to 2.4.x but there would be quite a few dependencies to pick up so how we proceed will depend on the timeline for your move to 2.5.x</p> <p>Regards</p> <p>Peter</p> <p>We will need to take several test shots between now putting 2.5 into production, but I think it's reasonable for us to target that (and avoid the work of backporting). We haven't seen this in production, yet.</p> <p>I guess my only question is:<br/> If we hit this in production, would a reboot and re-mount hit the problem again? Or is it intermittent?</p> <p>Only reboot or re-mount but without 6669 patch applied can NOT resolve the issue, even though it may work for a while, you still will hit it again some time later.</p> <p>Matt, how about your process? what do you want us to do next step?</p> <p>backport patch for 2.4.x is on <a href="http://review.whamcloud.com/#/c/10061/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10061/</a></p> <p>Matt, </p> <p>Since the fix is in b2_5 and later releases and there is a patch for b2_4, should we close this ticket or is there something else you need from us?</p> <p>Thanks</p> <p>Yes, we can close it. Thanks.</p> Related LU-3335 Development Rank 1|hzwilb: Rank (Obsolete) 13287