https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14470 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[15965.280047] LustreError: 23882:0:(llite_lib.c:1442:ll_update_lsm_md()) lustre: [0x200008107:0x10653:0x0] dir layout mismatch: [15965.283219] LustreError: 23882:0:(lustre_lmv.h:99:lsm_md_dump()) magic 0xcd20cd0 stripe count 2 master mdt 0 hash type 0x2 version 0 migrate offset 0 migrate hash 0x0 pool [15965.287312] LustreError: 23882:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[0] [0x20000dec0:0x7:0x0] [15965.289569] LustreError: 23882:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[1] [0x24000e690:0x7:0x0] [15965.291807] LustreError: 23882:0:(lustre_lmv.h:99:lsm_md_dump()) magic 0xcd20cd0 stripe count 2 master mdt 0 hash type 0x2 version 0 migrate offset 0 migrate hash 0x0 pool [15965.295841] LustreError: 23882:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[0] [0x20000e690:0x3:0x0] [15965.298063] LustreError: 23882:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[1] [0x24000ee60:0x1:0x0] [15965.310206] LustreError: 23884:0:(lustre_lmv.h:99:lsm_md_dump()) magic 0xcd20cd0 stripe count 2 master mdt 0 hash type 0x2 version 0 migrate offset 0 migrate hash 0x0 pool [15965.314355] LustreError: 23884:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[0] [0x20000dec0:0x7:0x0] [15965.316652] LustreError: 23884:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[1] [0x24000e690:0x7:0x0] [15965.318881] LustreError: 23884:0:(lustre_lmv.h:99:lsm_md_dump()) magic 0xcd20cd0 stripe count 2 master mdt 0 hash type 0x2 version 0 migrate offset 0 migrate hash 0x0 pool [15965.322888] LustreError: 23884:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[0] [0x20000e690:0x3:0x0] [15965.325121] LustreError: 23884:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[1] [0x24000ee60:0x1:0x0] [15965.340329] LustreError: 23886:0:(lustre_lmv.h:99:lsm_md_dump()) magic 0xcd20cd0 stripe count 2 master mdt 0 hash type 0x2 version 0 migrate offset 0 migrate hash 0x0 pool [15965.344411] LustreError: 23886:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[0] [0x20000dec0:0x7:0x0] [15965.346655] LustreError: 23886:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[1] [0x24000e690:0x7:0x0] [15965.348866] LustreError: 23886:0:(lustre_lmv.h:99:lsm_md_dump()) magic 0xcd20cd0 stripe count 2 master mdt 0 hash type 0x2 version 0 migrate offset 0 migrate hash 0x0 pool [15965.352827] LustreError: 23886:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[0] [0x20000e690:0x3:0x0] [15965.355133] LustreError: 23886:0:(lustre_lmv.h:103:lsm_md_dump()) stripe[1] [0x24000ee60:0x1:0x0] [15965.357439] LustreError: 23886:0:(llite_lib.c:2471:ll_prep_inode()) new_inode -fatal: rc -22 </pre> </div></div> <p>Create request is replayed but MDS creates striped directory shards with new fids, so client fails layout check.<br/> It can be reproduced by recovery-mds-scale or custom test case, I'll attach it later.</p> <p>For me it looks like a design flaw. Client should replay create request with previously allocated fids and MDS should recreate directory shards using client fids.</p> LU-14470

striped directory layout mismatch after failover

Bug Critical Open Unresolved Lai Siyao Andriy Skulysh Tue, 23 Feb 2021 20:35:03 +0000 Thu, 28 Sep 2023 07:59:38 +0000 Lustre 2.16.0 0 5 <p>Andriy Skulysh (c17819@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/41731" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41731</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14470" title="striped directory layout mismatch after failover" class="issue-link" data-issue-key="LU-14470">LU-14470</a> test: striped dir layout mismatch after failover<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 48d6ff02041dd6d8124d364f784e31ec6517bf7c</p> <p>Dual or multiple MDT failure recovery is hard to support. </p> <p>For striped directory creation, the stripe FIDs are allocated by MDTs, and these FIDs may not be usable in recovery(meta sequence not allocated yet), or they may be used by other objects (FID allocated by other operations in recovery).</p> <p>But if we don't reuse the FIDs, there are other problems, even if we update directory layout after replay (instead of reporting error currently), the following touch under this striped directory can't be replayed because its parent has changed.</p> <p>Some distributed transaction replays need all the information stored in update logs, because these transactions allocated FID on MDTs, e.g. striped directory creation and directory migration (to striped directory). Such operation replay can't be done from client side.</p> <p>One way to improve this is to store update logs on more MDTs than those involved, e.g., if a striped directory is created on MDT0 and MDT1, it also stores update logs on MDT2 (this can be configured, and may have more backups), so upon recovery, all the information can be obtained from MDT2 and successfully replayed.</p> <p>Since the FIDs for the MDT directory stripes are allocated by the MDS, I think there are two options here:</p> <ul class="alternate" type="square"> <li>have the client replay the mkdir with the original MDT FIDs in the LMV xattr, the same way that client replay of regular files includes the OST object FIDs. That ensures the recreated directory is exactly the same as the original and avoids failures in later replay operations. I think this would be my preferred solution, but it potentially exposes the internal filesystem structure to inconsistency if the client specifies wrong FIDs during replay (though see <a href="https://jira.whamcloud.com/browse/LU-15250" title="RPC Replay Signature" class="issue-link" data-issue-key="LU-15250">LU-15250</a>).</li> <li>the client should use the FID of the <b>master</b> directory for the parent FID (not the shard FID) when replaying the file create, and then lookup the shard FID from the directory during replay. The client doing mkdir originally selected the master directory FID and will use it for replay, so it will not change. That works around the problem with the shard FIDs changing during replay, but may still have other problems later if the shard FIDs are also used for other operations (though they might be fixed similarly to use the master FID). I think this adds complexity to the client replay process, but has the benefit that the client doesn't know as much detail about the shard layout, which <em>might</em> be helpful in the future.</li> </ul> <p>I do not think that storing the update logs on other MDTs is a good solution to this problem for several reasons:</p> <ul class="alternate" type="square"> <li>this makes distributed transactions even slower than they currently are, since it involves at least one extra MDT in each mkdir</li> <li>there may not even <b>be</b> an additional MDT to store the update log (e.g. in a two-MDT system)</li> <li>the additional MDT used to store the update log may <b>also</b> fail at the same time (e.g. power loss of server rack), so is not really solving the problem<br/> It is better to have a solution that more closely follows the existing recovery mechanisms that Lustre uses (e.g. replay with LMV xattr) instead of adding a different mechanism.</li> </ul> <p>After some tests, I found this should be a test script issue. The sync-on-lock-cancel mechanism has eliminated dependency between striped directory creation and sub file creation:<br/> 1. striped directory creation will hold UPDATE locks of all stripes after creation, the 1st stripe UPDATE lock is a local lock, while 2nd to the last stripe UPDATE locks are remote locks.<br/> 2. sub file creation will getattr on parent directory first, which will revoke UPDATE locks held held in step 1.<br/> 3. since the 1st stripe UPDATE lock is a local lock, it will be silently dropped, while for 2nd to last stripe UPDATE locks, they are remote locks, this will trigger commit-on-sharing, which guarantees 2nd to last stripes creation and update logs are committed to disk, after this the parent directory can always be recovered from update logs.<br/> 4. if all involved MDTs are rebooted at this moment, the striped directory will be recovered from update logs.<br/> 5. if sub file is located on the same MDT of the parent directory (also the 1st stripe), its creation will be replayed after parent directory replay, it's okay; otherwise the sub file creation replay may be before parent directory replay because they are replayed on different MDTs, however since 2nd to last stripe has been committed to disk already, even if the parent directory is not recovered from update logs yet, the according stripe is there already, and sub file creation replay will succeed.</p> <p>In summary, if two operation replays have dependency, but they may be replayed in random order (because they are replayed on different MDTs), this dependency should be eliminated (either by commit-on-sharing or by sync-on-lock-cancel). And in this case, there is no need to implement striped directory client replay, because if there is no subsequent operation depending on it, it can be a fresh new creation, while if there exists dependency, this dependency will be eliminated by sync-on-lock-cancel.</p> <p>BTW, I'm afraid this can't be tested with replay_barrier() which will fail step 2, so the test failed in the end. This should probably be tested on real machines.</p> <p>I will add a test to verify sync-on-lock-cancel is triggered for sub file creation under striped directory.</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45870" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45870</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14470" title="striped directory layout mismatch after failover" class="issue-link" data-issue-key="LU-14470">LU-14470</a> test: add striped directory creation test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9d88d35f954ff6cf93d8f53894a54e05fe39efad</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47385" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47385</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14470" title="striped directory layout mismatch after failover" class="issue-link" data-issue-key="LU-14470">LU-14470</a> dne: striped mkdir replay<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f7d4bb673b3cabce4e70029e2ff50f56e94a40ca</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/47385/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/47385/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14470" title="striped directory layout mismatch after failover" class="issue-link" data-issue-key="LU-14470">LU-14470</a> dne: striped mkdir replay by client request<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a2e997f0bed0ae4cfdcf6d73f8a79e3d23d28a2f</p> Related LU-16336 Development Rank 1|i01ngf: Rank (Obsolete) 9223372036854775807 Severity