[LU-2888] After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.4.0, Lustre 2.1.6
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.4
• Environment: before upgrade, server and client: 2.1.4 RHEL6; after upgrade, server and client: lustre-master build# 1270 RHEL6
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Sarah Liu

Description

Here are what I did:
1. format the system as 2.1.4 and then upgrade to 2.4, success.
2. showdown the filesystem and disable quota
3. downgrade the system to 2.1.4 again, when mount MDS, hit following errors

Here is the console of MDS:

Lustre: DEBUG MARKER: == upgrade-downgrade End == 18:53:45 (1362020025)
LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
Lustre: MGS MGS started
Lustre: 7888:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 7306ea48-8511-52b2-40cf-6424fc417e41@0@lo t0 exp (null) cur 1362020029 last 0
Lustre: MGC10.10.4.132@tcp: Reactivating import
Lustre: MGS: Logs for fs lustre were removed by user request.  All servers must be restarted in order to regenerate the logs.
Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Setting parameter lustre-clilov.lov.stripesize in log lustre-client
Lustre: Enabling ACL
Lustre: Enabling user_xattr
LustreError: 7901:0:(osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: 
LustreError: 7901:0:(osd_handler.c:2343:osd_index_try()) LBUG
Pid: 7901, comm: llog_process_th

Message from
Call Trace:
 syslogd@fat-amd [<ffffffffa03797f5>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
-1 at Feb 27 18: [<ffffffffa0379e07>] lbug_with_loc+0x47/0xb0 [libcfs]
53:49 ...
 ker [<ffffffffa0d6bd74>] osd_index_try+0x84/0x540 [osd_ldiskfs]
nel:LustreError: [<ffffffffa04c1dfe>] dt_try_as_dir+0x3e/0x60 [obdclass]
 7901:0:(osd_han [<ffffffffa0c5eb3a>] orph_index_init+0x6a/0x1e0 [mdd]
dler.c:2343:osd_ [<ffffffffa0c6ec45>] mdd_prepare+0x1d5/0x640 [mdd]
index_try()) ASS [<ffffffffa0ccd23c>] ? mdt_process_config+0x6c/0x1030 [mdt]
ERTION( dt_objec [<ffffffffa0da0499>] cmm_prepare+0x39/0xe0 [cmm]
t_exists(dt) ) f [<ffffffffa0ccfd7d>] mdt_device_alloc+0xe0d/0x2190 [mdt]
ailed: 

Me [<ffffffffa04bdeff>] ? keys_fill+0x6f/0x1a0 [obdclass]
ssage from syslo [<ffffffffa04a2c87>] obd_setup+0x1d7/0x2f0 [obdclass]
gd@fat-amd-1 at  [<ffffffffa048ef3b>] ? class_new_export+0x72b/0x960 [obdclass]
Feb 27 18:53:49  [<ffffffffa04a2fa8>] class_setup+0x208/0x890 [obdclass]
...
 kernel:Lu [<ffffffffa04aac6c>] class_process_config+0xc3c/0x1c30 [obdclass]
streError: 7901: [<ffffffffa037a993>] ? cfs_alloc+0x63/0x90 [libcfs]
0:(osd_handler.c [<ffffffffa04a5813>] ? lustre_cfg_new+0x353/0x7e0 [obdclass]
:2343:osd_index_ [<ffffffffa04acd0b>] class_config_llog_handler+0x9bb/0x1610 [obdclass]
try()) LBUG
 [<ffffffffa0637e3b>] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc]
 [<ffffffffa0478098>] llog_process_thread+0x888/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c14a>] child_rip+0xa/0x20
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c140>] ? child_rip+0x0/0x20

Kernel panic - not syncing: LBUG
Pid: 7901, comm: llog_process_th Not tainted 2.6.32-279.14.1.el6_lustre.x86_64 #1
Call Trace:

 [<ffffffff814fdcba>] ? panic+0xa0/0x168
Message from sy [<ffffffffa0379e5b>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
slogd@fat-amd-1  [<ffffffffa0d6bd74>] ? osd_index_try+0x84/0x540 [osd_ldiskfs]
at Feb 27 18:53: [<ffffffffa04c1dfe>] ? dt_try_as_dir+0x3e/0x60 [obdclass]
49 ...
 kernel [<ffffffffa0c5eb3a>] ? orph_index_init+0x6a/0x1e0 [mdd]
:Kernel panic -  [<ffffffffa0c6ec45>] ? mdd_prepare+0x1d5/0x640 [mdd]
not syncing: LBU [<ffffffffa0ccd23c>] ? mdt_process_config+0x6c/0x1030 [mdt]
G
 [<ffffffffa0da0499>] ? cmm_prepare+0x39/0xe0 [cmm]
 [<ffffffffa0ccfd7d>] ? mdt_device_alloc+0xe0d/0x2190 [mdt]
 [<ffffffffa04bdeff>] ? keys_fill+0x6f/0x1a0 [obdclass]
 [<ffffffffa04a2c87>] ? obd_setup+0x1d7/0x2f0 [obdclass]
 [<ffffffffa048ef3b>] ? class_new_export+0x72b/0x960 [obdclass]
 [<ffffffffa04a2fa8>] ? class_setup+0x208/0x890 [obdclass]
 [<ffffffffa04aac6c>] ? class_process_config+0xc3c/0x1c30 [obdclass]
 [<ffffffffa037a993>] ? cfs_alloc+0x63/0x90 [libcfs]
 [<ffffffffa04a5813>] ? lustre_cfg_new+0x353/0x7e0 [obdclass]
 [<ffffffffa04acd0b>] ? class_config_llog_handler+0x9bb/0x1610 [obdclass]
 [<ffffffffa0637e3b>] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc]
 [<ffffffffa0478098>] ? llog_process_thread+0x888/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c14a>] ? child_rip+0xa/0x20
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c140>] ? child_rip+0x0/0x20
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu

Issue Links

• duplicates: LU-3574 MDT cannot sync with OST after upgrade from 2.1 to 2.4 then downgrade
• is related to: LU-3267 cl_echo_object_find()) ASSERTION( ostid_seq(&lsm->lsm_wire.lw_object_oi) == FID_SEQ_ECHO ) failed:
• is related to: LU-2684 convert ost_id to lu_fid for FID_SEQ_NORMAL objects

Comments

[Sarah Liu]

I hit this issue when downgrade from 2.5 to 2.3.0, will create a new issue for that.

[Cory Spitz]

I also think that 2.3->2.4->2.3 upgrade & downgrade should be supported. But this bug occurs there as well. Is the appropriate response to reopen this ticket and mark it as affecting 2.3 (and I suppose 2.2 too)? Or should we make a clone?

Also, I think it is a pretty bad story that we'd have to get a fix onto b2_3 to make the upgrade/downgrade scenario work. Because even if we get the existing patch landed to b2_3 you can't revert to 2.3.0.

[Cory Spitz]

Yes, I think that we should support 1.8.x->2.4->1.8.9-wc1 upgrade & downgrade. Of course, w/o dirdata and large_xattr, etc. used.