[LU-8514] tgt_main.c:121:tgt_cancel_slc_locks()) ASSERTION( lock->l_client_cookie != 0 ) failed

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.9.0
• Type: Bug
• Priority: Critical
• Reporter: Christopher Morrone
• Assignee: Lai Siyao
• Resolution: Fixed
• Labels: llnl
• Severity: 3

Description

Running Lustre version 2.8.0_0.0.llnlpreview.33 (see repo lustre-release-fe-llnl), we hit the following assertion on one of our MDS nodes (out of 16) in our 2.8 DNE testbed:

2016-08-18 05:45:32 [47956.538091] LustreError: 17552:0:(llog_cat.c:385:llog_cat_current_log()) lquake-MDT0004-osd: next log does not exist!
2016-08-18 05:45:32 [47956.550273] LustreError: 17552:0:(update_trans.c:1006:top_trans_stop()) lquake-MDT0004-osd: write updates failed: rc = -14
2016-08-18 05:45:32 [47956.592024] LustreError: 15805:0:(tgt_main.c:121:tgt_cancel_slc_locks()) ASSERTION( lock->l_client_cookie != 0 ) failed:
2016-08-18 05:45:32 [47956.604585] LustreError: 15805:0:(tgt_main.c:121:tgt_cancel_slc_locks()) LBUG
2016-08-18 05:45:32 [47956.612773] Pid: 15805, comm: tx_commit_cb

Note that the "next log does not exist!" message was introduced by the patch from LU-7800, Change-Id: I2343023c1f3109c077c98d78d3669377d95ed42f, Patch-Set: 6.

Comments

[Peter Jones]

Lai

Could you please look into this one?

Thanks

Peter

[Di Wang]

Lai, It looks like we should not save slc locks if transaction fails?

static void mdt_save_remote_lock(struct mdt_thread_info *info,
                                 struct lustre_handle *h, enum ldlm_mode mode,
                                 int decref)
{
        ENTRY;

        if (lustre_handle_is_used(h)) {
                if (decref || !info->mti_has_trans ||
                    !(mode & (LCK_PW | LCK_EX))) {    ----> check failure transaction here.
                        ldlm_lock_decref_and_cancel(h, mode);
                } else {
                        struct ldlm_lock *lock = ldlm_handle2lock(h);
                        struct ptlrpc_request *req = mdt_info_req(info);

                        LASSERT(req != NULL);
                        tgt_save_slc_lock(lock, req->rq_transno);
                        ldlm_lock_decref(h, mode);
                }
                h->cookie = 0ull;
        }

        EXIT;
}

[Lai Siyao]

IMHO the cause if transaction failure is not checked in mdt handlers, do you know why it's always ignored?

[Christopher Morrone]

We hit this a couple of more times. Now it is hitting almost as soon as the MDS is brought back up. This is currently our worst blocker in testing on the DNE testbed.

[Di Wang]

IMHO the cause if transaction failure is not checked in mdt handlers, do you know why it's always ignored?

Usually if the transaction fails, info->mti_has_trans should not be set, but unfortunately for this case, the local transaction succeeds (i.e. info->mti_has_trans is set), but the remote transaction fails, so I think you need add another flag or sth to check this failure.

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: http://review.whamcloud.com/22071
Subject: LU-8514 mdd: transaction failure should be checked
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b1d88d35ee5b156aed9a3c4ab61fedc8f1821257

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/22071/
Subject: LU-8514 mdd: transaction failure should be checked
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e1ace3751f9add26b3f01aad9c278b6bfca8f739

[Peter Jones]

Landed for 2.9