https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4653 Lustre <p>Hit following LBUG when running rolling upgrade testing. Test steps are<br/> 1. setup the system with 2.5.0<br/> 2. keep the system on and upgrade OSS to lustre-master build #1876(teg-2.5.55)<br/> 3. when mounting OST, MDS reboot</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. LustreError: 2152:0:(lustre_fid.h:725:lu_fid_diff()) ASSERTION( fid_seq(fid1) == fid_seq(fid2) ) failed: fid1:[0x100000000:0x21:0x0], fid2:[0x100000001:0x0:0x0] LustreError: 2152:0:(lustre_fid.h:725:lu_fid_diff()) LBUG Pid: 2152, comm: osp-pre-0 Call Trace: [&lt;ffffffffa01ff895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa01ffe97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0af896d&gt;] osp_precreate_cleanup_orphans+0x10fd/0x1130 [osp] [&lt;ffffffffa0498161&gt;] ? import_at_get_index+0xb1/0xf0 [ptlrpc] [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0afac6f&gt;] osp_precreate_thread+0x20f/0x1b00 [osp] [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0afaa60&gt;] ? osp_precreate_thread+0x0/0x1b00 [osp] [&lt;ffffffff81096a36&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG Pid: 2152, comm: osp-pre-0 Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1 Call Trace: [&lt;ffffffff8150de58&gt;] ? panic+0xa7/0x16f [&lt;ffffffffa01ffeeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa0af896d&gt;] ? osp_precreate_cleanup_orphans+0x10fd/0x1130 [osp] [&lt;ffffffffa0498161&gt;] ? import_at_get_index+0xb1/0xf0 [ptlrpc] [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0afac6f&gt;] ? osp_precreate_thread+0x20f/0x1b00 [osp] [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0afaa60&gt;] ? osp_precreate_thread+0x0/0x1b00 [osp] [&lt;ffffffff81096a36&gt;] ? kthread+0x96/0xa0 [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Linux version 2.6.32-358.18.1.el6_lustre.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Fri Oct 11 16:41:53 PDT 2013 Command line: ro root=UUID=dec021fd-a287-4254-8c2b-0a004dfdde46 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD console=ttyS0,115200 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off memmap=exactmap memmap=538K@64K memmap=132562K@49690K elfcorehdr=182252K memmap=64K$0K memmap=38K$602K memmap=104K$920K memmap=8K$3668600K memmap=72K#3668608K memmap=184K#3668680K memmap=263296K$3668864K memmap=2048K$4192256K KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD Centaur CentaurHauls BIOS-provided physical RAM map: </pre> </div></div> before upgrade:<br/> server and client are running 2.5.0<br/> <br/> upgrade OSS to lustre-master build # 1876<br/> MDS and clients are still running 2.5.0 LU-4653

Hit LBUG ASSERTION( fid_seq(fid1) == fid_seq(fid2) ) failed after upgrade OST from 2.5.0 to 2.6

Bug Blocker Resolved Fixed Di Wang Sarah Liu mn4 Thu, 20 Feb 2014 08:01:03 +0000 Wed, 28 May 2014 07:09:00 +0000 Thu, 24 Apr 2014 13:40:54 +0000 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.4.3 Lustre 2.6.0 Lustre 2.5.2 0 5 <p>Di,<br/> Can you please have a look at this one and comment?<br/> Thank you!</p> <p>Hmm, probably caused by this commit</p> <p>commit da12d3ba35bbb86c8e5860a5ed161a55f01b69d5<br/> Author: Fan Yong &lt;fan.yong@intel.com&gt;<br/> Date: Sun Nov 24 16:59:00 2013 +0800</p> <p> <a href="https://jira.whamcloud.com/browse/LU-3569" title="Use real OST index as ostid_to_fid() parameter instead of always &quot;0&quot;" class="issue-link" data-issue-key="LU-3569"><del>LU-3569</del></a> ofd: packing ost_idx in IDIF</p> <p> For a normal FID, we can know on which target the related object<br/> is allocated via querying FLDB; but it is not true for an IDIF.</p> <p> To locate the OST via the given IDIF, when the IDIF is generated,<br/> we pack the OST index in it. Then for any given FID, in spite of<br/> it is a normal FID or not, we has the method to know which target<br/> it belongs to. That is useful for LFSCK.</p> <p> For old IDIF, the OST index is not part of the IDIF, means that<br/> different OSTs may have the same IDIFs, that may cause the IFID<br/> in LMA does not match the read FID. Under such case, we need to<br/> make some compatible check to avoid to trigger unexpected.</p> <p> tgt_validate_obdo() converts the ostid contained in the RPC body<br/> to fid and changes the "struct ost_id" union, then the users can<br/> access ost_id::oi_fid directly without call ostid_to_fid() again.</p> <p> It also contains some other fixing and cleanup.</p> <p> Test-Parameters: testlist=sanity-scrub<br/> Signed-off-by: wang di &lt;di.wang@intel.com&gt;<br/> Signed-off-by: Fan Yong &lt;fan.yong@intel.com&gt;<br/> Change-Id: I228f2f6cd9310193a1724046cee15e3b2103c8e2<br/> Reviewed-on: <a href="http://review.whamcloud.com/7053" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7053</a><br/> Tested-by: Jenkins<br/> Tested-by: Maloo &lt;hpdd-maloo@intel.com&gt;<br/> Reviewed-by: Andreas Dilger &lt;andreas.dilger@intel.com&gt;<br/> Reviewed-by: Alex Zhuravlev &lt;alexey.zhuravlev@intel.com&gt;</p> <p><a href="http://review.whamcloud.com/9560" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9560</a></p> <p>Hit this LBUG and MDS reboot </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fat-amd-1.lab.whamcloud.com login: Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. LustreError: 2165:0:(lustre_fid.h:725:lu_fid_diff()) ASSERTION( fid_seq(fid1) == fid_seq(fid2) ) failed: fid1:[0x100000000:0x21:0x0], fid2:[0x100000001:0x0:0x0] LustreError: 2165:0:(lustre_fid.h:725:lu_fid_diff()) LBUG Pid: 2165, comm: osp-pre-0 Call Trace: [&lt;ffffffffa0216895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0216e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0b1da3d&gt;] osp_precreate_cleanup_orphans+0x10fd/0x1130 [osp] [&lt;ffffffffa04b0d81&gt;] ? import_at_get_index+0xb1/0xf0 [ptlrpc] [&lt;ffffffff81065df0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0b1fd4f&gt;] osp_precreate_thread+0x20f/0x1b00 [osp] [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 [&lt;ffffffff81065df0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0b1fb40&gt;] ? osp_precreate_thread+0x0/0x1b00 [osp] [&lt;ffffffff8109aee6&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 [&lt;ffffffff8109ae50&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG Pid: 2165, comm: osp-pre-0 Not tainted 2.6.32-431.5.1.el6_lustre.x86_64 #1 Call Trace: [&lt;ffffffff81527983&gt;] ? panic+0xa7/0x16f [&lt;ffffffffa0216eeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa0b1da3d&gt;] ? osp_precreate_cleanup_orphans+0x10fd/0x1130 [osp] [&lt;ffffffffa04b0d81&gt;] ? import_at_get_index+0xb1/0xf0 [ptlrpc] [&lt;ffffffff81065df0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0b1fd4f&gt;] ? osp_precreate_thread+0x20f/0x1b00 [osp] [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 [&lt;ffffffff81065df0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0b1fb40&gt;] ? osp_precreate_thread+0x0/0x1b00 [osp] [&lt;ffffffff8109aee6&gt;] ? kthread+0x96/0xa0 [&lt;ffffffff8100c20a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffff8109ae50&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 Initializing cgroup subsys cpuset Initializing cgroup subsys cpu </pre> </div></div> <p>Ah, it is rolling upgrade. MDS still run with 2.5 version. so we need port <a href="http://review.whamcloud.com/9560" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9560</a> to 2.5.</p> <p><a href="http://review.whamcloud.com/#/c/10058" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10058</a> b2_5<br/> <a href="http://review.whamcloud.com/10059" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10059</a> b2_4</p> <p>I tried the patch for b2_5, got following error in the recovery stage after OSS upgrade to 2.6, MDS reboot again</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fat-amd-1.lab.whamcloud.com login: root Password: Lustre: lustre-MDT0000: Recovery over after 0:10, of 2 clients 2 recovered and 0 were evicted. LustreError: 2166:0:(osp_precreate.c:476:osp_precreate_send()) ASSERTION( osp_fid_diff(fid, &amp;d-&gt;opd_pre_used_fid) &gt; 0 ) failed: reply fid [0x100000001:0x0:0x0] pre used fid [0x100000000:0x1e8:0x0] LustreError: 2166:0:(osp_precreate.c:476:osp_precreate_send()) LBUG Pid: 2166, comm: osp-pre-0 Call Trace: [&lt;ffffffffa0209895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0209e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0b119d7&gt;] osp_precreate_send+0x1a47/0x1b00 [osp] [&lt;ffffffffa0491304&gt;] ? lustre_msg_set_timeout+0x74/0xc0 [ptlrpc] [&lt;ffffffffa0b11f79&gt;] osp_precreate_thread+0x4e9/0xc50 [osp] [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 [&lt;ffffffff81065df0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0b11a90&gt;] ? osp_precreate_thread+0x0/0xc50 [osp] Last login: Wed [&lt;ffffffff8109aee6&gt;] kthread+0x96/0xa0 Apr 23 17:31:38 [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 on ttyS0 [&lt;ffffffff8109ae50&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG Pid: 2166, comm: osp-pre-0 Not tainted 2.6.32-431.5.1.el6_lustre.x86_64 #1 Call Trace: [&lt;ffffffff81527983&gt;] ? panic+0xa7/0x16f [&lt;ffffffffa0209eeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa0b119d7&gt;] ? osp_precreate_send+0x1a47/0x1b00 [osp] [&lt;ffffffffa0491304&gt;] ? lustre_msg_set_timeout+0x74/0xc0 [ptlrpc] [&lt;ffffffffa0b11f79&gt;] ? osp_precreate_thread+0x4e9/0xc50 [osp] [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 [&lt;ffffffff81065df0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0b11a90&gt;] ? osp_precreate_thread+0x0/0xc50 [osp] [&lt;ffffffff8109aee6&gt;] ? kthread+0x96/0xa0 [&lt;ffffffff8100c20a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffff8109ae50&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 Initializing cgroup subsys cpuset Initializing cgroup subsys cpu </pre> </div></div> <p>Patch landed to Master. Patches for other branches will be tracked outside of this ticket.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: lustre-MDT0000: Recovery over after 0:10, of 2 clients 2 recovered and 0 were evicted. LustreError: 2166:0:(osp_precreate.c:476:osp_precreate_send()) ASSERTION( osp_fid_diff(fid, &amp;d-&gt;opd_pre_used_fid) &gt; 0 ) failed: reply fid [0x100000001:0x0:0x0] pre used fid [0x100000000:0x1e8:0x0] LustreError: 2166:0:(osp_precreate.c:476:osp_precreate_send()) LBUG Pid: 2166, comm: osp-pre-0 </pre> </div></div> <p>Hmm, this is different issue here, it seems MDT get a lower FID( &lt; its last precreate used FID) from OST during recovery, which is wrong. Though this ASSERT might be improper here. But anyway probably a new ticket?</p> <p>Create a new ticket in <a href="https://jira.whamcloud.com/browse/LU-4957" title=":osp_precreate_send()) ASSERTION( osp_fid_diff(fid, &amp;d-&gt;opd_pre_used_fid) &gt; 0 ) failed: reply fid [0x100000001:0x0:0x0] pre used fid [0x100000000:0x1e8:0x0]" class="issue-link" data-issue-key="LU-4957"><del>LU-4957</del></a></p> <p>Di, look at the details:</p> <p><span class="error">&#91;0x100000001:0x0:0x0&#93;</span> pre used fid <span class="error">&#91;0x100000000:0x1e8:0x0&#93;</span> LustreError: 2166:0:(osp_precreate.c:476:osp_precreate_send()) LBUG</p> <p>so, it got a new sequence?</p> <p>Alex: 0x100000001 is the same sequence as 0x100000000, and we just embedded ost_index here. or you mean sth else? btw: probably discussed this in 4957?</p> <p>Sara, could you try to reproduce this and grab a full debug log please? you can attach that to <a href="https://jira.whamcloud.com/browse/LU-4957" title=":osp_precreate_send()) ASSERTION( osp_fid_diff(fid, &amp;d-&gt;opd_pre_used_fid) &gt; 0 ) failed: reply fid [0x100000001:0x0:0x0] pre used fid [0x100000000:0x1e8:0x0]" class="issue-link" data-issue-key="LU-4957"><del>LU-4957</del></a></p> <p>Alex, ok, will do it now</p> <p>Patch landed for 2.5.2.</p> Related LU-4957 Development Rank 1|hzwffj: Rank (Obsolete) 12727 Severity