[LU-2888] After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Major
Fix Version/s: Lustre 2.4.0, Lustre 2.1.6
Affects Version/s: Lustre 2.4.0, Lustre 2.1.4
Labels:
None
Environment:
before upgrade, server and client: 2.1.4 RHEL6
after upgrade, server and client: lustre-master build# 1270 RHEL6

Severity:
3
Rank (Obsolete):
6970

Description

Here are what I did:
1. format the system as 2.1.4 and then upgrade to 2.4, success.
2. showdown the filesystem and disable quota
3. downgrade the system to 2.1.4 again, when mount MDS, hit following errors

Here is the console of MDS:

Lustre: DEBUG MARKER: == upgrade-downgrade End == 18:53:45 (1362020025)
LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.
LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: 
Lustre: MGS MGS started
Lustre: 7888:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 7306ea48-8511-52b2-40cf-6424fc417e41@0@lo t0 exp (null) cur 1362020029 last 0
Lustre: MGC10.10.4.132@tcp: Reactivating import
Lustre: MGS: Logs for fs lustre were removed by user request.  All servers must be restarted in order to regenerate the logs.
Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000
Lustre: Setting parameter lustre-clilov.lov.stripesize in log lustre-client
Lustre: Enabling ACL
Lustre: Enabling user_xattr
LustreError: 7901:0:(osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: 
LustreError: 7901:0:(osd_handler.c:2343:osd_index_try()) LBUG
Pid: 7901, comm: llog_process_th

Message from
Call Trace:
 syslogd@fat-amd [<ffffffffa03797f5>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
-1 at Feb 27 18: [<ffffffffa0379e07>] lbug_with_loc+0x47/0xb0 [libcfs]
53:49 ...
 ker [<ffffffffa0d6bd74>] osd_index_try+0x84/0x540 [osd_ldiskfs]
nel:LustreError: [<ffffffffa04c1dfe>] dt_try_as_dir+0x3e/0x60 [obdclass]
 7901:0:(osd_han [<ffffffffa0c5eb3a>] orph_index_init+0x6a/0x1e0 [mdd]
dler.c:2343:osd_ [<ffffffffa0c6ec45>] mdd_prepare+0x1d5/0x640 [mdd]
index_try()) ASS [<ffffffffa0ccd23c>] ? mdt_process_config+0x6c/0x1030 [mdt]
ERTION( dt_objec [<ffffffffa0da0499>] cmm_prepare+0x39/0xe0 [cmm]
t_exists(dt) ) f [<ffffffffa0ccfd7d>] mdt_device_alloc+0xe0d/0x2190 [mdt]
ailed: 

Me [<ffffffffa04bdeff>] ? keys_fill+0x6f/0x1a0 [obdclass]
ssage from syslo [<ffffffffa04a2c87>] obd_setup+0x1d7/0x2f0 [obdclass]
gd@fat-amd-1 at  [<ffffffffa048ef3b>] ? class_new_export+0x72b/0x960 [obdclass]
Feb 27 18:53:49  [<ffffffffa04a2fa8>] class_setup+0x208/0x890 [obdclass]
...
 kernel:Lu [<ffffffffa04aac6c>] class_process_config+0xc3c/0x1c30 [obdclass]
streError: 7901: [<ffffffffa037a993>] ? cfs_alloc+0x63/0x90 [libcfs]
0:(osd_handler.c [<ffffffffa04a5813>] ? lustre_cfg_new+0x353/0x7e0 [obdclass]
:2343:osd_index_ [<ffffffffa04acd0b>] class_config_llog_handler+0x9bb/0x1610 [obdclass]
try()) LBUG
 [<ffffffffa0637e3b>] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc]
 [<ffffffffa0478098>] llog_process_thread+0x888/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c14a>] child_rip+0xa/0x20
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c140>] ? child_rip+0x0/0x20

Kernel panic - not syncing: LBUG
Pid: 7901, comm: llog_process_th Not tainted 2.6.32-279.14.1.el6_lustre.x86_64 #1
Call Trace:

 [<ffffffff814fdcba>] ? panic+0xa0/0x168
Message from sy [<ffffffffa0379e5b>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
slogd@fat-amd-1  [<ffffffffa0d6bd74>] ? osd_index_try+0x84/0x540 [osd_ldiskfs]
at Feb 27 18:53: [<ffffffffa04c1dfe>] ? dt_try_as_dir+0x3e/0x60 [obdclass]
49 ...
 kernel [<ffffffffa0c5eb3a>] ? orph_index_init+0x6a/0x1e0 [mdd]
:Kernel panic -  [<ffffffffa0c6ec45>] ? mdd_prepare+0x1d5/0x640 [mdd]
not syncing: LBU [<ffffffffa0ccd23c>] ? mdt_process_config+0x6c/0x1030 [mdt]
G
 [<ffffffffa0da0499>] ? cmm_prepare+0x39/0xe0 [cmm]
 [<ffffffffa0ccfd7d>] ? mdt_device_alloc+0xe0d/0x2190 [mdt]
 [<ffffffffa04bdeff>] ? keys_fill+0x6f/0x1a0 [obdclass]
 [<ffffffffa04a2c87>] ? obd_setup+0x1d7/0x2f0 [obdclass]
 [<ffffffffa048ef3b>] ? class_new_export+0x72b/0x960 [obdclass]
 [<ffffffffa04a2fa8>] ? class_setup+0x208/0x890 [obdclass]
 [<ffffffffa04aac6c>] ? class_process_config+0xc3c/0x1c30 [obdclass]
 [<ffffffffa037a993>] ? cfs_alloc+0x63/0x90 [libcfs]
 [<ffffffffa04a5813>] ? lustre_cfg_new+0x353/0x7e0 [obdclass]
 [<ffffffffa04acd0b>] ? class_config_llog_handler+0x9bb/0x1610 [obdclass]
 [<ffffffffa0637e3b>] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc]
 [<ffffffffa0478098>] ? llog_process_thread+0x888/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c14a>] ? child_rip+0xa/0x20
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffffa0477810>] ? llog_process_thread+0x0/0xd00 [obdclass]
 [<ffffffff8100c140>] ? child_rip+0x0/0x20
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

2.4-start-mdt.log.tar.bz2
169 kB
09/Apr/13 4:49 PM
upgrade-ls.log.tar.bz2
160 kB
12/Apr/13 1:22 AM

Issue Links

duplicates

LU-3574 MDT cannot sync with OST after upgrade from 2.1 to 2.4 then downgrade

Resolved

is related to

LU-3267 cl_echo_object_find()) ASSERTION( ostid_seq(&lsm->lsm_wire.lw_object_oi) == FID_SEQ_ECHO ) failed:

Resolved

is related to

LU-2684 convert ost_id to lu_fid for FID_SEQ_NORMAL objects

Resolved

Activity

[LU-2888] After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed

Zhenyu Xu added a comment - 18/Apr/13 6:28 AM

landed on b2_1 for 2.1.6

Zhenyu Xu added a comment - 18/Apr/13 6:28 AM landed on b2_1 for 2.1.6

Sarah Liu added a comment - 16/Apr/13 5:58 PM - edited

Sorry, I was downgraded the system to b2_1 by mistake, 1.8.9<->2.4 doesn't have this issue.

Sarah Liu added a comment - 16/Apr/13 5:58 PM - edited Sorry, I was downgraded the system to b2_1 by mistake, 1.8.9<->2.4 doesn't have this issue.

Zhenyu Xu added a comment - 16/Apr/13 6:09 AM

Do we support 2.x to 1.8 downgrade?

Zhenyu Xu added a comment - 16/Apr/13 6:09 AM Do we support 2.x to 1.8 downgrade?

Sarah Liu added a comment - 16/Apr/13 5:38 AM - edited

Please also make changes to b1_8, hit the same LBUG after downgrade from the latest tag-2.3.64 to 1.8.9

Lustre: Enabling user_xattr
LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: 
LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) LBUG
Pid: 27458, comm: llog_process_th

Sarah Liu added a comment - 16/Apr/13 5:38 AM - edited Please also make changes to b1_8, hit the same LBUG after downgrade from the latest tag-2.3.64 to 1.8.9 Lustre: Enabling user_xattr LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) LBUG Pid: 27458, comm: llog_process_th

Jodi Levi (Inactive) added a comment - 15/Apr/13 12:56 PM

Reducing 2.4 blocker, but keeping open until lands on b2_1.

Jodi Levi (Inactive) added a comment - 15/Apr/13 12:56 PM Reducing 2.4 blocker, but keeping open until lands on b2_1.

Zhenyu Xu added a comment - 15/Apr/13 12:43 PM

http://review.whamcloud.com/#change,5731 hasn't landed on b2_1 yet, need that to be claimed to be fixed.

Zhenyu Xu added a comment - 15/Apr/13 12:43 PM http://review.whamcloud.com/#change,5731 hasn't landed on b2_1 yet, need that to be claimed to be fixed.

Jodi Levi (Inactive) added a comment - 15/Apr/13 12:39 PM

Landed for 2.4

Jodi Levi (Inactive) added a comment - 15/Apr/13 12:39 PM Landed for 2.4

Jodi Levi (Inactive) added a comment - 12/Apr/13 8:38 PM - edited

Change/6034 and 6037 have been merged into http://review.whamcloud.com/#change,6044

Jodi Levi (Inactive) added a comment - 12/Apr/13 8:38 PM - edited Change/6034 and 6037 have been merged into http://review.whamcloud.com/#change,6044

Zhenyu Xu added a comment - 12/Apr/13 8:41 AM

yes, with http://review.whamcloud.com/5731 on b2_1 and http://review.whamcloud.com/6034 and http://review.whamcloud.com/6037 on master, the downgrade and upgrade test passed with no noise.

Zhenyu Xu added a comment - 12/Apr/13 8:41 AM yes, with http://review.whamcloud.com/5731 on b2_1 and http://review.whamcloud.com/6034 and http://review.whamcloud.com/6037 on master, the downgrade and upgrade test passed with no noise.

Di Wang added a comment - 12/Apr/13 7:22 AM

http://review.whamcloud.com/#change,6037 Bobi: please check this one. Thanks!

Di Wang added a comment - 12/Apr/13 7:22 AM http://review.whamcloud.com/#change,6037 Bobi: please check this one. Thanks!

Di Wang added a comment - 12/Apr/13 3:27 AM

Hmm, the problem is that in 2.1, we define the lsm/lmm in this

struct lov_mds_md_v1 {            /* LOV EA mds/wire data (little-endian) */
        __u32 lmm_magic;          /* magic number = LOV_MAGIC_V1 */
        __u32 lmm_pattern;        /* LOV_PATTERN_RAID0, LOV_PATTERN_RAID1 */
        __u64 lmm_object_id;      /* LOV object ID */
        __u64 lmm_object_seq;     /* LOV object seq number */
        __u32 lmm_stripe_size;    /* size of stripe in bytes */
        __u32 lmm_stripe_count;   /* num stripes in use for this object */
        struct lov_ost_data_v1 lmm_objects[0]; /* per-stripe data */
};

But lmm_object_seq/lmm_object_id is actually normal MDT FIDS, i.e. lmm_object_id/lmm_object_seq will be f_oid/normal_seq, and when unpack lmm to lsm on 2.4,

it will use ostid_le_to_cpu()

static inline void ostid_le_to_cpu(struct ost_id *src_oi,
                                   struct ost_id *dst_oi)
{
        if (fid_seq_is_mdt0(ostid_seq(src_oi))) {
                dst_oi->oi.oi_id = le64_to_cpu(src_oi->oi.oi_id);
                dst_oi->oi.oi_seq = le64_to_cpu(src_oi->oi.oi_seq);
        } else {
                fid_le_to_cpu(&dst_oi->oi_fid, &src_oi->oi_fid);
        }
}

And treat the ostid as normal FID, which cause the problem.

Sigh, it seems we do not have better way to convert this special ostid to the real FID.

Di Wang added a comment - 12/Apr/13 3:27 AM Hmm, the problem is that in 2.1, we define the lsm/lmm in this struct lov_mds_md_v1 { /* LOV EA mds/wire data (little-endian) */ __u32 lmm_magic; /* magic number = LOV_MAGIC_V1 */ __u32 lmm_pattern; /* LOV_PATTERN_RAID0, LOV_PATTERN_RAID1 */ __u64 lmm_object_id; /* LOV object ID */ __u64 lmm_object_seq; /* LOV object seq number */ __u32 lmm_stripe_size; /* size of stripe in bytes */ __u32 lmm_stripe_count; /* num stripes in use for this object */ struct lov_ost_data_v1 lmm_objects[0]; /* per-stripe data */ }; But lmm_object_seq/lmm_object_id is actually normal MDT FIDS, i.e. lmm_object_id/lmm_object_seq will be f_oid/normal_seq, and when unpack lmm to lsm on 2.4, it will use ostid_le_to_cpu() static inline void ostid_le_to_cpu(struct ost_id *src_oi, struct ost_id *dst_oi) { if (fid_seq_is_mdt0(ostid_seq(src_oi))) { dst_oi->oi.oi_id = le64_to_cpu(src_oi->oi.oi_id); dst_oi->oi.oi_seq = le64_to_cpu(src_oi->oi.oi_seq); } else { fid_le_to_cpu(&dst_oi->oi_fid, &src_oi->oi_fid); } } And treat the ostid as normal FID, which cause the problem. Sigh, it seems we do not have better way to convert this special ostid to the real FID.

People

Assignee:: Zhenyu Xu

Reporter:: Sarah Liu

Votes:: 0 Vote for this issue

Watchers:: 14 Start watching this issue

Dates

Created:: 27/Feb/13 10:02 PM

Updated:: 25/Jul/13 8:07 PM

Resolved:: 18/Apr/13 6:28 AM