[LU-5651] ASSERTION( req->rq_export->exp_lock_replay_needed ) failed

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Labels: patch
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Andriy Skulysh

Description

Client doesn't restore import state correctly
on reconnect during replay. It resends lock replay
when final ping was queued by server.
Server fails with "target_queue_recovery_request())
ASSERTION( req->rq_export->exp_lock_replay_needed ) failed"

Solution is to add imp_replay_state to store last replay state.
During reconnect imp_state is restored from imp_replay_state.

Issue Links

• is duplicated by: LU-5719 target_queue_recovery_request() ASSERTION failed
• is related to: LU-5287 (ldlm_lib.c:2253:target_queue_recovery_request()) ASSERTION( req->rq_export->exp_lock_replay_needed ) failed

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12942/
Subject: LU-5651 test: run replay-single test 93 only when supported.
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: afde9f17260650d0cb80d53613fb5afda0a39384

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: http://review.whamcloud.com/12942
Subject: LU-5651 test: run replay-single test 93 only when supported.
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7bc6d9b8c2e09027444b31c40dd320e244c2412d

[James A Simmons]

I believe the replay-single test needs to be updated to test for lustre versions so interop testing passes.

[James Nunez (Inactive)]

Patch for b2_5 at http://review.whamcloud.com/#/c/12163/

The master patch has landed. Is there more work needed to complete this ticket or should it be closed?

[Jason Hill (Inactive)]

ORNL hit this issue today in production after upgrading to Lustre 2.5.2 on both server and client.

[Niu Yawei (Inactive)]

Ah, my mistake, the code has been changed by LU-793, so client can reconnect now. I'll review the patch soon, thanks for your explanation.

[Andriy Skulysh]

I don't understand why reconnect should be rejected. The only check for inflight RPC on a export is

no_export:
                OBD_FAIL_TIMEOUT(OBD_FAIL_TGT_DELAY_CONNECT, 2 * obd_timeout);
        } else if (req->rq_export == NULL &&
		   atomic_read(&export->exp_rpc_count) > 0) {
                LCONSOLE_WARN("%s: Client %s (at %s) refused connection, "
                              "still busy with %d references\n",
                              target->obd_name, cluuid.uuid,
                              libcfs_nid2str(req->rq_peer.nid),
			      atomic_read(&export->exp_refcount));
                GOTO(out, rc = -EBUSY);

but reconnect request has valid export handle.

[Niu Yawei (Inactive)]

no. request reaches timeout on reply, client reconnects

Could you explain how can client1 reconnect when the final ping is in queue (server would reject the reconnect because the export has inflight RPC)? Is there defect in reconnect path?

[Andriy Skulysh]

no. request reaches timeout on reply, client reconnects

[Niu Yawei (Inactive)]

step by step explanation:
1) server goes to recovery stage 2 (lock replay) with at least 2 clients.
2) client1 and client 2 send lock relay,
3) client1 sends final ping
4) server queues final ping from client1 and set exp_lock_replay_needed to 0
5) client2 still in lock replay stage (waits for lock replies form server)
6) client1 reconnects

Is the final ping from client1 still in queue? If it's still in queue, client1 can't reconnect because there is inflgiht RPC; If the final ping has been processed, the exp_in_recovery should have been cleared.

[Andriy Skulysh]

step by step explanation:
1) server goes to recovery stage 2 (lock replay) with at least 2 clients.
2) client1 and client 2 send lock relay,
3) client1 sends final ping
4) server queues final ping from client1 and set exp_lock_replay_needed to 0
5) client2 still in lock replay stage (waits for lock replies form server)
6) client1 reconnects
7) client1 replays locks from the beginning
8) assertion fails