[LU-14642] transfer layout version to OST objects in layout change Created: 26/Apr/21 Updated: 13/Jan/23 Resolved: 17/Sep/22 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | Lustre 2.16.0 |
| Type: | Bug | Priority: | Critical |
| Reporter: | Zhenyu Xu | Assignee: | Zhenyu Xu |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||||||
| Severity: | 3 | ||||||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||||||
| Epic Link: | FLR tech debt review | ||||||||||||
| Description |
|
There are cases that layout version has not been transferred to OST object after mirror extend/split/resync which makes following sync hang. OFD will compare the layout version from client with on-disk object's in ofd_verify_layout_version(), as the client's version increased with mirror extend/split/resync, the sync IO will keep loop with EINPROGRESS reply from the OFD.
|
| Comments |
| Comment by Zhenyu Xu [ 28/Apr/21 ] |
|
Add a FLR mode in fsx.c to test various IO upon mirrored file. |
| Comment by Gerrit Updater [ 28/Apr/21 ] |
|
Bobi Jam (bobijam@hotmail.com) uploaded a new patch: https://review.whamcloud.com/43472 |
| Comment by Gerrit Updater [ 28/Apr/21 ] |
|
Bobi Jam (bobijam@hotmail.com) uploaded a new patch: https://review.whamcloud.com/43473 |
| Comment by Gerrit Updater [ 02/Jun/21 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/43472/ |
| Comment by Andreas Dilger [ 20/Oct/21 ] |
|
Bobijam, is there a reason why the MDS must update the layout version on the OST object before the client RPC can complete? If the client has a newer layout version than the OST, it would make sense that the client knows the new layout still includes the OST object, and the OST could just update the layout version directly. This is very different from the case where the client sends an RPC with old layout version and may need to update the layout in order to generate the correct RPC. That may mean the client is flushing old data without the layout lock being revoked, and it needs to contact the MDS to get the new file layout. Not only would this avoid the requirement for the MDS to send an urgent RPC to the OSS to avoid the client IO being blocked, but it seems like the MDS may not need to send any RPC at all, or at least not a distributed transaction RPC, if the client is still actively writing to the object. That would also avoid the race condition between the file layout being changed on the MDT and it being updated on all of the OST objects (which may be up to 2000 objects for a very wide striped layout). If a client sends an RPC with a new layout version to the OST first, then it can immediately update the object version and continue, unlike the current case of the client being blocked and waiting for a long time for the OST object version to be updated. |
| Comment by Zhenyu Xu [ 20/Oct/21 ] |
|
sound reasonable, OST only allows newer version write from the client and that does not need MDS's notification to OST about the layout version change. |
| Comment by Chris Hunter (Inactive) [ 29/Oct/21 ] |
|
Thanks for the update bobijam |
| Comment by Gerrit Updater [ 03/Nov/21 ] |
|
"Bobi Jam <bobijam@hotmail.com>" uploaded a new patch: https://review.whamcloud.com/45443 |
| Comment by Gerrit Updater [ 30/May/22 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/43473/ |
| Comment by Gerrit Updater [ 08/Jun/22 ] |
|
"Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/47567 |
| Comment by Gerrit Updater [ 11/Jul/22 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/47567/ |
| Comment by Colin Faber [ 25/Jul/22 ] |
|
what's going on with this fix? |
| Comment by Gerrit Updater [ 17/Sep/22 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/45443/ |
| Comment by Peter Jones [ 17/Sep/22 ] |
|
Landed for 2.16 |