[LU-2888] After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Major
Fix Version/s: Lustre 2.4.0, Lustre 2.1.6
Affects Version/s: Lustre 2.4.0, Lustre 2.1.4
Labels:
None
Environment:
before upgrade, server and client: 2.1.4 RHEL6
after upgrade, server and client: lustre-master build# 1270 RHEL6

Severity:
3
Rank (Obsolete):
6970

Description

Here are what I did:
1. format the system as 2.1.4 and then upgrade to 2.4, success.
2. showdown the filesystem and disable quota
3. downgrade the system to 2.1.4 again, when mount MDS, hit following errors

Here is the console of MDS:

Lustre: DEBUG MARKER: == upgrade-downgrade End == 18:53:45 (1362020025)
LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
Lustre: MGS MGS started
Lustre: 7888:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 7306ea48-8511-52b2-40cf-6424fc417e41@0@lo t0 exp (null) cur 1362020029 last 0
Lustre: MGC10.10.4.132@tcp: Reactivating import
Lustre: MGS: Logs for fs lustre were removed by user request.  All servers must be restarted in order to regenerate the logs.
Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Setting parameter lustre-clilov.lov.stripesize in log lustre-client
Lustre: Enabling ACL
Lustre: Enabling user_xattr
LustreError: 7901:0:(osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: 
LustreError: 7901:0:(osd_handler.c:2343:osd_index_try()) LBUG
Pid: 7901, comm: llog_process_th

Message from
Call Trace:
 syslogd@fat-amd [<ffffffffa03797f5>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
-1 at Feb 27 18: [<ffffffffa0379e07>] lbug_with_loc+0x47/0xb0 [libcfs]
53:49 ...
 ker [<ffffffffa0d6bd74>] osd_index_try+0x84/0x540 [osd_ldiskfs]
nel:LustreError: [<ffffffffa04c1dfe>] dt_try_as_dir+0x3e/0x60 [obdclass]
 7901:0:(osd_han [<ffffffffa0c5eb3a>] orph_index_init+0x6a/0x1e0 [mdd]
dler.c:2343:osd_ [<ffffffffa0c6ec45>] mdd_prepare+0x1d5/0x640 [mdd]
index_try()) ASS [<ffffffffa0ccd23c>] ? mdt_process_config+0x6c/0x1030 [mdt]
ERTION( dt_objec [<ffffffffa0da0499>] cmm_prepare+0x39/0xe0 [cmm]
t_exists(dt) ) f [<ffffffffa0ccfd7d>] mdt_device_alloc+0xe0d/0x2190 [mdt]
ailed: 

Me [<ffffffffa04bdeff>] ? keys_fill+0x6f/0x1a0 [obdclass]
ssage from syslo [<ffffffffa04a2c87>] obd_setup+0x1d7/0x2f0 [obdclass]
gd@fat-amd-1 at  [<ffffffffa048ef3b>] ? class_new_export+0x72b/0x960 [obdclass]
Feb 27 18:53:49  [<ffffffffa04a2fa8>] class_setup+0x208/0x890 [obdclass]
...
 kernel:Lu [<ffffffffa04aac6c>] class_process_config+0xc3c/0x1c30 [obdclass]
streError: 7901: [<ffffffffa037a993>] ? cfs_alloc+0x63/0x90 [libcfs]
0:(osd_handler.c [<ffffffffa04a5813>] ? lustre_cfg_new+0x353/0x7e0 [obdclass]
:2343:osd_index_ [<ffffffffa04acd0b>] class_config_llog_handler+0x9bb/0x1610 [obdclass]
try()) LBUG
 [<ffffffffa0637e3b>] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc]
 [<ffffffffa0478098>] llog_process_thread+0x888/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c14a>] child_rip+0xa/0x20
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c140>] ? child_rip+0x0/0x20

Kernel panic - not syncing: LBUG
Pid: 7901, comm: llog_process_th Not tainted 2.6.32-279.14.1.el6_lustre.x86_64 #1
Call Trace:

 [<ffffffff814fdcba>] ? panic+0xa0/0x168
Message from sy [<ffffffffa0379e5b>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
slogd@fat-amd-1  [<ffffffffa0d6bd74>] ? osd_index_try+0x84/0x540 [osd_ldiskfs]
at Feb 27 18:53: [<ffffffffa04c1dfe>] ? dt_try_as_dir+0x3e/0x60 [obdclass]
49 ...
 kernel [<ffffffffa0c5eb3a>] ? orph_index_init+0x6a/0x1e0 [mdd]
:Kernel panic -  [<ffffffffa0c6ec45>] ? mdd_prepare+0x1d5/0x640 [mdd]
not syncing: LBU [<ffffffffa0ccd23c>] ? mdt_process_config+0x6c/0x1030 [mdt]
G
 [<ffffffffa0da0499>] ? cmm_prepare+0x39/0xe0 [cmm]
 [<ffffffffa0ccfd7d>] ? mdt_device_alloc+0xe0d/0x2190 [mdt]
 [<ffffffffa04bdeff>] ? keys_fill+0x6f/0x1a0 [obdclass]
 [<ffffffffa04a2c87>] ? obd_setup+0x1d7/0x2f0 [obdclass]
 [<ffffffffa048ef3b>] ? class_new_export+0x72b/0x960 [obdclass]
 [<ffffffffa04a2fa8>] ? class_setup+0x208/0x890 [obdclass]
 [<ffffffffa04aac6c>] ? class_process_config+0xc3c/0x1c30 [obdclass]
 [<ffffffffa037a993>] ? cfs_alloc+0x63/0x90 [libcfs]
 [<ffffffffa04a5813>] ? lustre_cfg_new+0x353/0x7e0 [obdclass]
 [<ffffffffa04acd0b>] ? class_config_llog_handler+0x9bb/0x1610 [obdclass]
 [<ffffffffa0637e3b>] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc]
 [<ffffffffa0478098>] ? llog_process_thread+0x888/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c14a>] ? child_rip+0xa/0x20
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c140>] ? child_rip+0x0/0x20
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

2.4-start-mdt.log.tar.bz2
169 kB
09/Apr/13 4:49 PM
upgrade-ls.log.tar.bz2
160 kB
12/Apr/13 1:22 AM

Issue Links

duplicates

LU-3574 MDT cannot sync with OST after upgrade from 2.1 to 2.4 then downgrade

Resolved

is related to

LU-3267 cl_echo_object_find()) ASSERTION( ostid_seq(&lsm->lsm_wire.lw_object_oi) == FID_SEQ_ECHO ) failed:

Resolved

is related to

LU-2684 convert ost_id to lu_fid for FID_SEQ_NORMAL objects

Resolved

Activity

[LU-2888] After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed

Sarah Liu added a comment - 25/Jul/13 8:06 PM - edited

I hit this issue when downgrade from 2.5 to 2.3.0, will create a new issue for that.

Sarah Liu added a comment - 25/Jul/13 8:06 PM - edited I hit this issue when downgrade from 2.5 to 2.3.0, will create a new issue for that.

Cory Spitz added a comment - 20/May/13 4:26 PM

I also think that 2.3->2.4->2.3 upgrade & downgrade should be supported. But this bug occurs there as well. Is the appropriate response to reopen this ticket and mark it as affecting 2.3 (and I suppose 2.2 too)? Or should we make a clone?

Also, I think it is a pretty bad story that we'd have to get a fix onto b2_3 to make the upgrade/downgrade scenario work. Because even if we get the existing patch landed to b2_3 you can't revert to 2.3.0.

Cory Spitz added a comment - 20/May/13 4:26 PM I also think that 2.3->2.4->2.3 upgrade & downgrade should be supported. But this bug occurs there as well. Is the appropriate response to reopen this ticket and mark it as affecting 2.3 (and I suppose 2.2 too)? Or should we make a clone? Also, I think it is a pretty bad story that we'd have to get a fix onto b2_3 to make the upgrade/downgrade scenario work. Because even if we get the existing patch landed to b2_3 you can't revert to 2.3.0.

Cory Spitz added a comment - 13/May/13 1:54 PM

Yes, I think that we should support 1.8.x->2.4->1.8.9-wc1 upgrade & downgrade. Of course, w/o dirdata and large_xattr, etc. used.

Cory Spitz added a comment - 13/May/13 1:54 PM Yes, I think that we should support 1.8.x->2.4->1.8.9-wc1 upgrade & downgrade. Of course, w/o dirdata and large_xattr, etc. used.

Zhenyu Xu added a comment - 18/Apr/13 6:28 AM

landed on b2_1 for 2.1.6

Zhenyu Xu added a comment - 18/Apr/13 6:28 AM landed on b2_1 for 2.1.6

Sarah Liu added a comment - 16/Apr/13 5:58 PM - edited

Sorry, I was downgraded the system to b2_1 by mistake, 1.8.9<->2.4 doesn't have this issue.

Sarah Liu added a comment - 16/Apr/13 5:58 PM - edited Sorry, I was downgraded the system to b2_1 by mistake, 1.8.9<->2.4 doesn't have this issue.

Zhenyu Xu added a comment - 16/Apr/13 6:09 AM

Do we support 2.x to 1.8 downgrade?

Zhenyu Xu added a comment - 16/Apr/13 6:09 AM Do we support 2.x to 1.8 downgrade?

Sarah Liu added a comment - 16/Apr/13 5:38 AM - edited

Please also make changes to b1_8, hit the same LBUG after downgrade from the latest tag-2.3.64 to 1.8.9

Lustre: Enabling user_xattr
LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: 
LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) LBUG
Pid: 27458, comm: llog_process_th

Sarah Liu added a comment - 16/Apr/13 5:38 AM - edited Please also make changes to b1_8, hit the same LBUG after downgrade from the latest tag-2.3.64 to 1.8.9 Lustre: Enabling user_xattr LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) LBUG Pid: 27458, comm: llog_process_th

Jodi Levi (Inactive) added a comment - 15/Apr/13 12:56 PM

Reducing 2.4 blocker, but keeping open until lands on b2_1.

Jodi Levi (Inactive) added a comment - 15/Apr/13 12:56 PM Reducing 2.4 blocker, but keeping open until lands on b2_1.

Zhenyu Xu added a comment - 15/Apr/13 12:43 PM

http://review.whamcloud.com/#change,5731 hasn't landed on b2_1 yet, need that to be claimed to be fixed.

Zhenyu Xu added a comment - 15/Apr/13 12:43 PM http://review.whamcloud.com/#change,5731 hasn't landed on b2_1 yet, need that to be claimed to be fixed.

Jodi Levi (Inactive) added a comment - 15/Apr/13 12:39 PM

Landed for 2.4

Jodi Levi (Inactive) added a comment - 15/Apr/13 12:39 PM Landed for 2.4

People

Assignee:: Zhenyu Xu

Reporter:: Sarah Liu

Votes:: 0 Vote for this issue

Watchers:: 14 Start watching this issue

Dates

Created:: 27/Feb/13 10:02 PM

Updated:: 25/Jul/13 8:07 PM

Resolved:: 18/Apr/13 6:28 AM