https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4878 Lustre <p>The following LBUG appeared at customer site, during the mount process on all OSS in lustre 2.4.3 version.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 12838:0:(fld_handler.c:172:fld_server_lookup()) ASSERTION(fld-&gt;lsf_control_exp ) failed: LustreError: 12838:0:(fld_handler.c:172:fld_server_lookup()) LBUG Pid: 12838, comm: mount.lustre Call Trace: libcfs_debug_dumpstack+0x55/0x80 [libcfs] lbug_with_loc+0x47/0xb0 [libcfs] fld_server_lookup+0x2f7/0x3d0 [fld] osd_fld_lookup+0x71/0x1d0 [osd_ldiskfs] osd_remote_fid+0x9a/0x280 [osd_ldiskfs] osd_index_ea_lookup+0521/0x850 [osd_ldiskfs] dt_lookup_dir+0x6f/0x130 [obdclass] llog_osd_open+0x485/0xc00 [obdclass] llog_open+0xba/0x2c0 [obdclass] mgc_process_log [mgc] mgc_process_config [mgc] lustre_process_log [obdclass] server_start_targets [obdclass] server_fill_super [obdclass] lustre_fill_super[obdclass] get_sb_nodev lustre_get_sb vfs_kern_mount do_kern_mount do_mount sys_mount system_call_fastpath </pre> </div></div> <p>This issue seems the same as <a href="https://jira.whamcloud.com/browse/LU-3126" title="conf-sanity test_41b: fld_server_lookup()) ASSERTION( fld-&gt;lsf_control_exp ) failed" class="issue-link" data-issue-key="LU-3126"><del>LU-3126</del></a> for which a patch has been landed in lustre 2.5. Unfortunately no patch has been provided for lustre 2.4 release.</p> LU-4878

fld_server_lookup() ASSERTION( fld->lsf_control_exp ) failed

Bug Major Resolved Fixed Bruno Faccini Gregoire Pichon mn4 Thu, 10 Apr 2014 14:07:58 +0000 Thu, 15 May 2014 14:03:10 +0000 Thu, 15 May 2014 14:03:10 +0000 Lustre 2.4.3 0 8 <p>Hello Gregoire,<br/> Did you mean that this same LBUG occured for all OSSs of the same FS at OSTs mount time ??<br/> And yes you are right <a href="https://jira.whamcloud.com/browse/LU-3126" title="conf-sanity test_41b: fld_server_lookup()) ASSERTION( fld-&gt;lsf_control_exp ) failed" class="issue-link" data-issue-key="LU-3126"><del>LU-3126</del></a> patch has not been back-ported to b2_4, but it is mainly because the issue was unlikely to happen ...</p> <p><a href="https://jira.whamcloud.com/browse/LU-3126" title="conf-sanity test_41b: fld_server_lookup()) ASSERTION( fld-&gt;lsf_control_exp ) failed" class="issue-link" data-issue-key="LU-3126"><del>LU-3126</del></a> backport to b2_4:<br/> <a href="http://review.whamcloud.com/9929" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9929</a></p> <p>Yes We meet the same LBUG on all OSS, with or without abort-recov<br/> when mounting all ost at the same time or when we try to mount<br/> manuely just one ost.<br/> and it s just with 2.4.3 with some additional patchs and works<br/> fine on 2.4.2 with some other additionnal patchs </p> <p>I have tested the patch #9929 posted in gerrit. Unfortunately the OSS still crashes when mounting OST.</p> <p>Here is the stack</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;3&gt;LustreError: 6577:0:(fld_handler.c:174:fld_server_lookup()) srv-fs_pv-OST0000: lookup 0x7d, but not connects to MDT0yet: rc = -5. &lt;3&gt;LustreError: 6577:0:(osd_handler.c:2135:osd_fld_lookup()) fs_pv-OST0000-osd: cannot find FLD range for 0x7d: rc = -5 &lt;3&gt;LustreError: 6577:0:(osd_handler.c:3344:osd_mdt_seq_exists()) fs_pv-OST0000-osd: Can not lookup fld for 0x7d &lt;0&gt;LustreError: 6577:0:(osd_handler.c:2651:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed: &lt;0&gt;LustreError: 6577:0:(osd_handler.c:2651:osd_object_ref_del()) LBUG &lt;4&gt;Pid: 6577, comm: mount.lustre &lt;4&gt; &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa0d57895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt; [&lt;ffffffffa0d57e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa04197e7&gt;] osd_object_ref_del+0x1e7/0x220 [osd_ldiskfs] &lt;4&gt; [&lt;ffffffffa0ec1fee&gt;] llog_osd_destroy+0x48e/0xb20 [obdclass] &lt;4&gt; [&lt;ffffffffa0e91d61&gt;] llog_destroy+0x51/0x170 [obdclass] &lt;4&gt; [&lt;ffffffffa0e96b34&gt;] llog_erase+0x1c4/0x1e0 [obdclass] &lt;4&gt; [&lt;ffffffffa0e97401&gt;] llog_backup+0x231/0x500 [obdclass] &lt;4&gt; [&lt;ffffffffa049ad66&gt;] mgc_process_log+0x1636/0x18f0 [mgc] &lt;4&gt; [&lt;ffffffffa049c514&gt;] mgc_process_config+0x594/0xed0 [mgc] &lt;4&gt; [&lt;ffffffffa0ede64c&gt;] lustre_process_log+0x25c/0xaa0 [obdclass] &lt;4&gt; [&lt;ffffffffa0f126d3&gt;] server_start_targets+0x1833/0x19c0 [obdclass] &lt;4&gt; [&lt;ffffffffa0f1340c&gt;] server_fill_super+0xbac/0x1660 [obdclass] &lt;4&gt; [&lt;ffffffffa0ee3d68&gt;] lustre_fill_super+0x1d8/0x530 [obdclass] &lt;4&gt; [&lt;ffffffff8118c7df&gt;] get_sb_nodev+0x5f/0xa0 &lt;4&gt; [&lt;ffffffffa0edb3b5&gt;] lustre_get_sb+0x25/0x30 [obdclass] &lt;4&gt; [&lt;ffffffff8118be3b&gt;] vfs_kern_mount+0x7b/0x1b0 &lt;4&gt; [&lt;ffffffff8118bfe2&gt;] do_kern_mount+0x52/0x130 &lt;4&gt; [&lt;ffffffff811acfeb&gt;] do_mount+0x2fb/0x930 &lt;4&gt; [&lt;ffffffff811ad6b0&gt;] sys_mount+0x90/0xe0 &lt;4&gt; [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b &lt;4&gt; </pre> </div></div> <p>Hello Gregoire,<br/> Thanks for the update.<br/> But concerning the new and different stack/traceback you reported, in my opinion it looks more like a different problem which has already been reported in <a href="https://jira.whamcloud.com/browse/LU-3915" title="After upgrade from 2.4.0 to 2.5, can not mount OST, (osd_handler.c:2668:osd_object_ref_del()) LBUG Pid: 9537, comm: mount.lustre" class="issue-link" data-issue-key="LU-3915"><del>LU-3915</del></a>.<br/> So you may now give a try to my b2_4 back-port, of master patch #7673 from <a href="https://jira.whamcloud.com/browse/LU-3915" title="After upgrade from 2.4.0 to 2.5, can not mount OST, (osd_handler.c:2668:osd_object_ref_del()) LBUG Pid: 9537, comm: mount.lustre" class="issue-link" data-issue-key="LU-3915"><del>LU-3915</del></a>, I just pushed at <a href="http://review.whamcloud.com/9958" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9958</a>.</p> <p>The LBUG I hit when testing path #9929 has the same stack trace than <a href="https://jira.whamcloud.com/browse/LU-3915" title="After upgrade from 2.4.0 to 2.5, can not mount OST, (osd_handler.c:2668:osd_object_ref_del()) LBUG Pid: 9537, comm: mount.lustre" class="issue-link" data-issue-key="LU-3915"><del>LU-3915</del></a>, which reports an OSS crash when mounting OSTs after an upgrade from 2.4.0 to 2.5.</p> <p>In my case, I was upgrading from 2.4.2 to 2.4.3 with a few additional patches including #5049 "<a href="https://jira.whamcloud.com/browse/LU-2059" title="mgc to backup configuration on osd-based llogs" class="issue-link" data-issue-key="LU-2059"><del>LU-2059</del></a> llog: MGC to use OSD API for backup logs".</p> <p>Patch #7673 "<a href="https://jira.whamcloud.com/browse/LU-3915" title="After upgrade from 2.4.0 to 2.5, can not mount OST, (osd_handler.c:2668:osd_object_ref_del()) LBUG Pid: 9537, comm: mount.lustre" class="issue-link" data-issue-key="LU-3915"><del>LU-3915</del></a> osd-ldiskfs: don't assert on possible upgrade" seems to fix upgrade issue caused by #5049. So I probably need to add #7673 in the list of patches on top of 2.4.3, do you agree ?</p> <p>You may have missed my previous update that already confirmed what you finally found!<br/> So yes, I agree that you can add #7673 or its back-port on top of your 2.4.3 version that also include #5049 ...</p> <p>Thanks for the backport Bruno. Our comments interleaved !</p> <p>I have tested a lustre version 2.4.3 with both additional patches</p> <ul class="alternate" type="square"> <li>#9929 <a href="https://jira.whamcloud.com/browse/LU-3126" title="conf-sanity test_41b: fld_server_lookup()) ASSERTION( fld-&gt;lsf_control_exp ) failed" class="issue-link" data-issue-key="LU-3126"><del>LU-3126</del></a> osd: remove fld lookup during configuration</li> <li>#9958 <a href="https://jira.whamcloud.com/browse/LU-4878" title="fld_server_lookup() ASSERTION( fld-&gt;lsf_control_exp ) failed" class="issue-link" data-issue-key="LU-4878"><del>LU-4878</del></a> osd-ldiskfs: don't assert on possible upgrade (backport of <a href="https://jira.whamcloud.com/browse/LU-3915" title="After upgrade from 2.4.0 to 2.5, can not mount OST, (osd_handler.c:2668:osd_object_ref_del()) LBUG Pid: 9537, comm: mount.lustre" class="issue-link" data-issue-key="LU-3915"><del>LU-3915</del></a>)</li> </ul> <p>The OSS is able to start without any problem. Filesystem is operational.</p> <p>I am now waiting for these patches to be fully approved and Maloo tested so they can be delivered to the customer.</p> <p>Hello Gregoire,<br/> I am not sure that my patch #9958 will finally be fully accepted+landed to b2_4 ... The main reason of this is that #5049 is itself still not in b2_4 and may be won't, so #9958 is not necessary then as Mike commented in patch with reason !! <br/> This points to some limit in the process where people use to decide to add more patches on top of releases we tested vs regressions and interoperability...</p> <p>Hello Bruno,</p> <p>Actually the patch #5049 "<a href="https://jira.whamcloud.com/browse/LU-2059" title="mgc to backup configuration on osd-based llogs" class="issue-link" data-issue-key="LU-2059"><del>LU-2059</del></a> llog: MGC to use OSD API for backup logs" has been integrated by Bull on top of release 2.4.x because the customer hit the LBUG ASSERTION(cli-&gt;cl_mgc_configs_dir) described in <a href="https://jira.whamcloud.com/browse/LU-2959" title="ASSERTION( cli-&gt;cl_mgc_configs_dir ) for 200 osts x 2 oss" class="issue-link" data-issue-key="LU-2959"><del>LU-2959</del></a>. As mentionned in that ticket, the LBUG is fixed by patch #5049.</p> <p>These problems occured in lustre 2.4.x release and need to be addressed.</p> <p>Gregoire, don't misunderstand me, I did not mean that you added patches without good reasons to do so, but only that doing so you fall back out from our regression/interop testing process.<br/> Concerning the fact you added #5049 due to <a href="https://jira.whamcloud.com/browse/LU-2959" title="ASSERTION( cli-&gt;cl_mgc_configs_dir ) for 200 osts x 2 oss" class="issue-link" data-issue-key="LU-2959"><del>LU-2959</del></a>, that may help for #5049 and #9958 to finally land ...</p> <p>Hello Gregoire,<br/> Since patch #9958 is planned for 2.4 integration, do you agree if we close/resolve this issue as fixed ? </p> <p>Hi Bruno,</p> <p>Yes this ticket can be closed since our tests have shown the issue is fixed with patches #9929 and #9958.<br/> I hope both patches are planned for integration in 2.4 if a new version is released.</p> <p>Yes this would be under consideration for 2.4.4.</p> Related LU-3126 LU-3915 Development Rank 1|hzwjqn: Rank (Obsolete) 13490 Severity