[LU-5651] ASSERTION( req->rq_export->exp_lock_replay_needed ) failed

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Type: Bug
• Priority: Minor
• Reporter: Andriy Skulysh
• Assignee: Niu Yawei (Inactive)
• Resolution: Fixed
• Labels: patch
• Severity: 3

Description

Client doesn't restore import state correctly
on reconnect during replay. It resends lock replay
when final ping was queued by server.
Server fails with "target_queue_recovery_request())
ASSERTION( req->rq_export->exp_lock_replay_needed ) failed"

Solution is to add imp_replay_state to store last replay state.
During reconnect imp_state is restored from imp_replay_state.

Issue Links

• is duplicated by: LU-5719 target_queue_recovery_request() ASSER...
• is related to: LU-5287 (ldlm_lib.c:2253:target_queue_recover...

Comments

[Andriy Skulysh]

patch: http://review.whamcloud.com/12015

[Niu Yawei (Inactive)]

This looks dup of LU-5287 and I posted a fix at: http://review.whamcloud.com/#/c/11871/

Client doesn't restore import state correctly
on reconnect during replay. It resends lock replay
when final ping was queued by server.
Server fails with "target_queue_recovery_request())
ASSERTION( req->rq_export->exp_lock_replay_needed ) failed"

If the final ping has been queued on server, the recovery on this export should be finished (exp_in_recovery == 0) when the resent lock replay reach server. I don't see why that could trigger the assertion, could you explain it in detail?

[Andriy Skulysh]

The recovery ends only on receiving and processing final pings from all clients. I can happen that server accepted final ping from client1 but waits for requests from client2 and client1 reconnects. This situation was simulated in test by adding timeout before processing final pings.

[Niu Yawei (Inactive)]

The recovery ends only on receiving and processing final pings from all clients.

I was saying recovery on this export is done. (exp_in_recovery == 0).

[Andriy Skulysh]

exp_in_recovery is zeroed during recovery final stage 3, client1 reconnects during lock replay stage 2.

[Andriy Skulysh]

step by step explanation:
1) server goes to recovery stage 2 (lock replay) with at least 2 clients.
2) client1 and client 2 send lock relay,
3) client1 sends final ping
4) server queues final ping from client1 and set exp_lock_replay_needed to 0
5) client2 still in lock replay stage (waits for lock replies form server)
6) client1 reconnects
7) client1 replays locks from the beginning
8) assertion fails

[Niu Yawei (Inactive)]

step by step explanation:
1) server goes to recovery stage 2 (lock replay) with at least 2 clients.
2) client1 and client 2 send lock relay,
3) client1 sends final ping
4) server queues final ping from client1 and set exp_lock_replay_needed to 0
5) client2 still in lock replay stage (waits for lock replies form server)
6) client1 reconnects

Is the final ping from client1 still in queue? If it's still in queue, client1 can't reconnect because there is inflgiht RPC; If the final ping has been processed, the exp_in_recovery should have been cleared.

[Andriy Skulysh]

no. request reaches timeout on reply, client reconnects

[Niu Yawei (Inactive)]

no. request reaches timeout on reply, client reconnects

Could you explain how can client1 reconnect when the final ping is in queue (server would reject the reconnect because the export has inflight RPC)? Is there defect in reconnect path?

[Andriy Skulysh]

I don't understand why reconnect should be rejected. The only check for inflight RPC on a export is

no_export:
                OBD_FAIL_TIMEOUT(OBD_FAIL_TGT_DELAY_CONNECT, 2 * obd_timeout);
        } else if (req->rq_export == NULL &&
		   atomic_read(&export->exp_rpc_count) > 0) {
                LCONSOLE_WARN("%s: Client %s (at %s) refused connection, "
                              "still busy with %d references\n",
                              target->obd_name, cluuid.uuid,
                              libcfs_nid2str(req->rq_peer.nid),
			      atomic_read(&export->exp_refcount));
                GOTO(out, rc = -EBUSY);

but reconnect request has valid export handle.

[Niu Yawei (Inactive)]

Ah, my mistake, the code has been changed by LU-793, so client can reconnect now. I'll review the patch soon, thanks for your explanation.

[Jason Hill (Inactive)]

ORNL hit this issue today in production after upgrading to Lustre 2.5.2 on both server and client.

[James Nunez (Inactive)]

Patch for b2_5 at http://review.whamcloud.com/#/c/12163/

The master patch has landed. Is there more work needed to complete this ticket or should it be closed?

[James A Simmons]

I believe the replay-single test needs to be updated to test for lustre versions so interop testing passes.

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: http://review.whamcloud.com/12942
Subject: LU-5651 test: run replay-single test 93 only when supported.
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7bc6d9b8c2e09027444b31c40dd320e244c2412d

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12942/
Subject: LU-5651 test: run replay-single test 93 only when supported.
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: afde9f17260650d0cb80d53613fb5afda0a39384

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12163/
Subject: LU-5651: ptlrpc: fix import state during replay
Project: fs/lustre-release
Branch: b2_5
Current Patch Set:
Commit: 5748379d28846c672f793ba1f8e143e63531dd05

[Jian Yu]

Patches landed to master and b2_5 branches.