[LU-6981] obd_last_committed is not updated in tgt_reply_data_init()

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.8.0
• Severity: 3
• Assignee: Di Wang
• Reporter: Di Wang

Description

In tgt_reply_data_init(), it should update obd_last_committed after analyzing those reply_data, similar as tgt_server_data_init(), otherwise obd_last_committed might be smaller than the real committed transno, and then cause recovery_thread stuck in check_for_next_transno(), which rely on the correct last committed transno.

Issue Links

• is related to: LU-6831 The ticket for tracking all DNE2 bugs
• is related to: LU-5319 Support multiple slots per client in last_rcvd file

Comments

[Joseph Gmitter (Inactive)]

Patch has landed for 2.8.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/15971/
Subject: LU-6981 target: update obd_last_committed
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f2cd05463982e2fc17d30c927065266850d55446

[Mikhail Pershin]

The queue_len is just an number of replays in the queue. If all nodes are online then it should be equal to obd_req_replay_clients when every client (and other MDS) will send replay. As I undestand the other MDT can send replays with DNE2, can't it? I mean that if check_for_next_transno() is stuck for some reason then this is not right, it wasn't designed to stuck and must go on at some point, e.g. client(MDT) eviction or so. And it obviously shouldn't be blocked by other MDT except the waiting for next replay. The obd_last_committed value is not the key here, it can cause just gaps more often but gaps may occur in normal recovery as well.

[Di Wang]

recovery will start from last_committed transno, and it will wait for the next transno to come. If we start from a smaller transno, then the next transno will never come (because it has been committed). Note: even with that waking for gap check, this will not work, because

        } else if (queue_len > 0 &&
                   queue_len == atomic_read(&obd->obd_req_replay_clients)) {

because obd->obd_req_replay_clients == client + MDS count, while queue_len might only be == client_count. especially when multiple MDTs failed at the same time.

[Mikhail Pershin]

while I agree with patch, I wonder now recovery stuck in check_for_next_transno()? Wrong last committed value shouldn't cause such thing, we might see just longer recovery maybe but not 'hung'. Could you provide more information here?

[Gerrit Updater]

wangdi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15971
Subject: LU-6981 target: update obd_last_committed
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a17ecaf704a41cc037dc974eec05e9e0ceeb25a0