[LU-7427] DNE3: multiple entries for BATCHID

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.8.0
• Labels: dne3
• Assignee: WC Triage
• Reporter: Di Wang (Inactive)

Description

In current DNE implementation (2.8.0), the DNE update records will be cancelled only if

1. all of updates of this operation have been committed disk.
2. all of operation with smaller batchid has been committed. And BATCHID has been updated.

If one operation fails or stucks somewhere, then all of update logs of the following operation will not be cancelled even all of its updates have been committed, which will cause a very long recovery time, because it needs to retrieve all of update log for recovery, which is observed in DNE failover soak-test.

So we can have multiple entries for batchid, i.e. records multiple batchids in BATCHID file, so even if one operation is stucked, it can still update the batchid, until all of entries are used. (similar as multiple last rcvd entry)

Issue Links

• is related to: LU-17818 LMR: Lustre Metadata Redundancy
• is related to: LU-12310 MDT Device-level Replication/Mirroring
• is related to: LU-7426 DNE3: improve llog format for remote update llog
• is related to: LU-4215 Some expected improvements for OUT

Comments

[Di Wang (Inactive)]

if updates are stored separately (like we discussed in the context of a different llog implementation), then we'd have to read only tiny descriptions like few dozen bytes / transaction.

Even if we only read tiny for each record, but we still need read these "useless" records from all of logs. My point is that we should delete these all-committed cancel logs asap to avoid long time recovery. The current single batchid is not smart enough.

[Alex Zhuravlev]

this is partly because we have to read all the updates from all the llogs. if updates are stored separately (like we discussed in the context of a different llog implementation), then we'd have to read only tiny descriptions like few dozen bytes / transaction.

[Di Wang (Inactive)]

hmm, I think the problem is that we have too much update log, which slow down the recovery speed.

I''d think you mean an additional index of committed-everywhere batchid's which you can clear by updating a single-value batchid.

I think we need a smarter batchid to remember the committed status, instead of current single-value batchid, which will easily block cancellation if one transaction in commit list is stucked.

[Alex Zhuravlev]

I think this wouldn't be a big issue if our logs are indices storing just a short description of transactions (like batchid -> [set of nodes + blob pointer]).
I'm not sure we really want an analogue of last_rcvd. I''d think you mean an additional index of committed-everywhere batchid's which you can clear by updating a single-value batchid.