https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3639 Lustre <p>This is the same error described in <a href="https://jira.whamcloud.com/browse/LU-2888" title="After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed" class="issue-link" data-issue-key="LU-2888"><del>LU-2888</del></a></p> <p>MDS console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. quota=off. Opts: LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. quota=off. Opts: LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. quota=off. Opts: Lustre: MGC10.10.4.132@tcp: Reactivating import Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs. Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Setting parameter lustre-clilov.lov.stripesize in log lustre-client LustreError: 8000:0:(osd_handler.c:2720:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: LustreError: 8000:0:(osd_handler.c:2720:osd_index_try()) LBUG Pid: 8000, comm: llog_process_th Call Trace: [&lt;ffffffffa0379905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0379f17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0e08735&gt;] osd_index_try+0x175/0x620 [osd_ldiskfs] [&lt;ffffffffa0842c08&gt;] fld_index_init+0x88/0x4d0 [fld] [&lt;ffffffffa084013d&gt;] ? fld_cache_init+0x14d/0x430 [fld] [&lt;ffffffffa083ba3e&gt;] fld_server_init+0x29e/0x450 [fld] [&lt;ffffffffa0d5c1b6&gt;] mdt_fld_init+0x126/0x430 [mdt] [&lt;ffffffffa0d61326&gt;] mdt_init0+0x8c6/0x23f0 [mdt] [&lt;ffffffffa0d5bf49&gt;] ? mdt_key_init+0x59/0x1a0 [mdt] [&lt;ffffffffa0d62f43&gt;] mdt_device_alloc+0xf3/0x220 [mdt] [&lt;ffffffffa04cb0d7&gt;] obd_setup+0x1d7/0x2f0 [obdclass] [&lt;ffffffffa04cb3f8&gt;] class_setup+0x208/0x890 [obdclass] [&lt;ffffffffa04d308c&gt;] class_process_config+0xc0c/0x1c30 [obdclass] [&lt;ffffffffa037abe0&gt;] ? cfs_alloc+0x30/0x60 [libcfs] [&lt;ffffffffa04cceb3&gt;] ? lustre_cfg_new+0x353/0x7e0 [obdclass] [&lt;ffffffffa04d515b&gt;] class_config_llog_handler+0x9bb/0x1610 [obdclass] [&lt;ffffffffa067530b&gt;] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc] [&lt;ffffffffa049e1f8&gt;] llog_process_thread+0x888/0xd00 [obdclass] [&lt;ffffffffa049d970&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [&lt;ffffffffa049d970&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffffa049d970&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG Pid: 8000, comm: llog_process_th Not tainted 2.6.32-279.5.1.el6_lustre.gb16fe80.x86_64 #1 Call Trace: [&lt;ffffffff814fd58a&gt;] ? panic+0xa0/0x168 [&lt;ffffffffa0379f6b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa0e08735&gt;] ? osd_index_try+0x175/0x620 [osd_ldiskfs] [&lt;ffffffffa0842c08&gt;] ? fld_index_init+0x88/0x4d0 [fld] [&lt;ffffffffa084013d&gt;] ? fld_cache_init+0x14d/0x430 [fld] [&lt;ffffffffa083ba3e&gt;] ? fld_server_init+0x29e/0x450 [fld] [&lt;ffffffffa0d5c1b6&gt;] ? mdt_fld_init+0x126/0x430 [mdt] [&lt;ffffffffa0d61326&gt;] ? mdt_init0+0x8c6/0x23f0 [mdt] [&lt;ffffffffa0d5bf49&gt;] ? mdt_key_init+0x59/0x1a0 [mdt] [&lt;ffffffffa0d62f43&gt;] ? mdt_device_alloc+0xf3/0x220 [mdt] [&lt;ffffffffa04cb0d7&gt;] ? obd_setup+0x1d7/0x2f0 [obdclass] [&lt;ffffffffa04cb3f8&gt;] ? class_setup+0x208/0x890 [obdclass] [&lt;ffffffffa04d308c&gt;] ? class_process_config+0xc0c/0x1c30 [obdclass] [&lt;ffffffffa037abe0&gt;] ? cfs_alloc+0x30/0x60 [libcfs] [&lt;ffffffffa04cceb3&gt;] ? lustre_cfg_new+0x353/0x7e0 [obdclass] [&lt;ffffffffa04d515b&gt;] ? class_config_llog_handler+0x9bb/0x1610 [obdclass] [&lt;ffffffffa067530b&gt;] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc] [&lt;ffffffffa049e1f8&gt;] ? llog_process_thread+0x888/0xd00 [obdclass] [&lt;ffffffffa049d970&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffffa049d970&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffffa049d970&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 Initializing cgroup subsys cpuset Initializing cgroup subsys cpu </pre> </div></div> before upgrade, server and client: 2.3.0<br/> after upgrade, server is 2.5, 2 clients are 2.5, 1 client is 2.3.0<br/> after downgrade, server and client: 2.3.0 LU-3639

After downgrade from 2.5 to 2.3.0, hit (osd_handler.c:2720:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed

Bug Trivial Resolved Won't Fix nasf Sarah Liu Thu, 25 Jul 2013 20:10:51 +0000 Fri, 23 Oct 2015 07:42:41 +0000 Fri, 23 Oct 2015 07:42:41 +0000 Lustre 2.3.0 0 9 <p>Is this the 2.3.0 server or IEEL?</p> <p>Hi Andreas:<br/> this is 2.3.0 server</p> <p>As Oleg suggested, I will rerun the test with IEEL server and see if it hit the same problem and will update this ticket when I have result.</p> <p>Sarah, any chance to run this test with IEEL?</p> <p>Hi Andreas,</p> <p>this is blocked by TEI-578</p> <p>Hmm, it seems FLD object(as a special FID) is not being inserted properly in 2.5, i.e. "fld" is not being insert as a special name. So when downgrade to 2.3, it will use name("fld") to locate FLD, but since the name is not being insert, so it caused the LBUG. Hmm, I saw osd_oi_lookup ignore LOCAL seq, Fan Yong could you please comment?</p> <p>Hi Di, you mean that in Lustre-2.5, the local file FLD, its name "fld", is not correctly inserted as special name, then when downgrade to Lustre-2.3, the old osd_oi_lookup() will try to lookup such special name, then failed?</p> <p>But in fact, in Lustre-2.5, we add the local objects both into OI tables and insert its special name as following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> osd_oi_insert(struct osd_thread_info *info, struct osd_device *osd, <span class="code-keyword">const</span> struct lu_fid *fid, <span class="code-keyword">const</span> struct osd_inode_id *id, handle_t *th, <span class="code-keyword">enum</span> oi_check_flags flags) { … rc = osd_oi_iam_refresh(info, osd_fid2oi(osd, fid), (<span class="code-keyword">const</span> struct dt_rec *)oi_id, (<span class="code-keyword">const</span> struct dt_key *)oi_fid, th, <span class="code-keyword">true</span>); … <span class="code-keyword">if</span> (unlikely(fid_seq(fid) == FID_SEQ_LOCAL_FILE)) rc = osd_obj_spec_insert(info, osd, fid, id, th); <span class="code-keyword">return</span> rc; } </pre> </div></div> <p>Here is the file one the ldiskfs partition created under Lustre-2.5:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># ls -a /mnt/mds1 ./ O/ changelog_catalog lfsck_bookmark lov_objseq oi.16.12 oi.16.17 oi.16.21 oi.16.26 oi.16.30 oi.16.35 oi.16.4 oi.16.44 oi.16.49 oi.16.53 oi.16.58 oi.16.62 quota_master/ ../ OI_scrub changelog_users lfsck_layout oi.16.0 oi.16.13 oi.16.18 oi.16.22 oi.16.27 oi.16.31 oi.16.36 oi.16.40 oi.16.45 oi.16.5 oi.16.54 oi.16.59 oi.16.63 quota_slave/ CATALOGS PENDING/ fld lfsck_namespace oi.16.1 oi.16.14 oi.16.19 oi.16.23 oi.16.28 oi.16.32 oi.16.37 oi.16.41 oi.16.46 oi.16.50 oi.16.55 oi.16.6 oi.16.7 seq_ctl CONFIGS/ REMOTE_PARENT_DIR/ hsm_actions lost+found/ oi.16.10 oi.16.15 oi.16.2 oi.16.24 oi.16.29 oi.16.33 oi.16.38 oi.16.42 oi.16.47 oi.16.51 oi.16.56 oi.16.60 oi.16.8 seq_srv NIDTBL_VERSIONS/ ROOT/ last_rcvd lov_objid oi.16.11 oi.16.16 oi.16.20 oi.16.25 oi.16.3 oi.16.34 oi.16.39 oi.16.43 oi.16.48 oi.16.52 oi.16.57 oi.16.61 oi.16.9 </pre> </div></div> <p>The "fld" is there. I am not sure whether it is your expected or not.</p> <p>oh, I mean osd_oi_lookup does not do special lookup for <span class="error">&#91;FID_SEQ_LOCAL_FILE, FLD_INDEX_OID, 0&#93;</span>"fld", and it might cause this problem.</p> <p>See this downgrade process, the filesystem <br/> 1. the FS is formatted in 2.3 first, fld is created with the name "fld"<br/> 2. then upgrade to 2.5, but in 2.5, it did not lookup "fld" in osd_oi_lookup, and instead creating a new one. (This is obviously wrong)<br/> 3. Then downgrade to 2.3, it can not find the old one. So I guess the fld might be deleted somehow because of 2.</p> <p>Here is one patch against b2_3 to resolve the issue:</p> <p><a href="http://review.whamcloud.com/10224" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10224</a></p> <p>Do we still maintain b2_3? If not, I will abandon the the patch <a href="http://review.whamcloud.com/10224" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10224</a> and close the ticket.</p> <p>Downgrade the priority since we have no detailed plan to land more patches to b2_3 recently.</p> <p>Since we will not land more patches to b2_3, close it.</p> Development Rank 1|hzvw87: Rank (Obsolete) 9373 Severity