https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12310 Lustre <p>During a discussion a lunch today at LUG we were talking about the work being done on DOM/DNE/PFL/FLR. We were also talking about Lustre becoming more than just a scratch file system and it occurred to me that one thing that really hampers that concept is the vulnerability of metadata in its present state. </p> <p>I don't recall metadata replication ever being mentioned but I think that I would be a valuable feature to be explored. </p> LU-12310

MDT Device-level Replication/Mirroring

New Feature Minor Open Unresolved Peter Jones Joe Mervini LMR Thu, 16 May 2019 19:22:06 +0000 Sat, 15 Apr 2023 22:40:35 +0000 0 9 <p>We've discussed this internally and worked up an initial design, but ended up deciding against that implementation. We have not had resources to work up a new design and work on the implementation.</p> <p>In the meantime, my recommendation would be to use MD-RAID/dm-mirror (or hardware-based mirror) for ldiskfs, or ZFS VDEV mirror to replicate the MDT storage across nodes (assuming non-shared storage is the goal), and continue to use failover to manage which MDS is exporting the MDT. The MD-RAID or ZFS VDEV would use 2x or 3x devices per mirror (each one in a separate node), and a network storage transport like NVMe-over-fabrics, SRP, or iSCSI. With modern storage transports, local storage and remote storage is equally fast, and the upper-layer RAID handles the replication and recovery of the storage devices in the same manner as local disks.</p> <p>This approach is described for ldiskfs at <a href="http://wiki.lustre.org/images/e/e7/LUG-2012-SRP_Mirroring-UFL.pdf" class="external-link" target="_blank" rel="nofollow noopener">High Availability Lustre Using SRP-mirrored LUNs</a>, and similar approaches have been discussed for ZFS, but I don't have a link handy.</p> <p>I think that parts of this work could be split into some smaller features/implementation tasks in order to reduce the amount of effort needed to get something usable out of the development:</p> <ul class="alternate" type="square"> <li>improvements to performance of distributed transactions (<a href="https://jira.whamcloud.com/browse/LU-7426" title="DNE3: Current llog format for remote update llog" class="issue-link" data-issue-key="LU-7426">LU-7426</a>) so that synchronous/ordered disk transactions are not needed. This would be very useful independent of MDT mirroring to improve creation of remote and striped directories, cross-MDT rename/link, etc.</li> <li>improve handling of distributed recovery when an MDT is offline (e.g. save transaction logs, don't block filesystem access for unrelated MDTs (<a href="https://jira.whamcloud.com/browse/LU-9206" title="DNE - allow partial access to striped dir if one of the MDTs is unavailable" class="issue-link" data-issue-key="LU-9206"><del>LU-9206</del></a> ++)</li> <li>fault-tolerance for services that run on MDT0000, such as the quota master, FLDB, MGT, etc.</li> <li>scalability of <tt>REMOTE_PARENT_DIR</tt> to allow handling more disconnected filesystem objects (<a href="https://jira.whamcloud.com/browse/LU-10329" title="DNE3: REMOTE_PARENT_DIR scalability" class="issue-link" data-issue-key="LU-10329">LU-10329</a>)</li> <li>mirroring of top-level directories in the filesystem (initiallly <tt>ROOT/</tt>, and then first level of subdirectories below it, etc.) so that the filesystem is "more" available if MDT0000 or other MDTs in a top-level striped directory are unavailable. This would <em>not</em> include mirroring of the regular inodes for files, only the directories themselves. Since the top-level directories are changed relatively less often than lower-level subdirectories, some extra overhead creating directories at this level is worthwhile for higher availability. <ul class="alternate" type="square"> <li>mirrored directories would be similar to striped directories, but each directory entry name could be looked up in at least two different directory shards (e.g. <tt>lmv_locate_tgt_by_name()</tt>, ...+1, ...+2), depending on replication level, allowing the target to be found even if one MDT is offline (<a href="https://jira.whamcloud.com/browse/LU-9206" title="DNE - allow partial access to striped dir if one of the MDTs is unavailable" class="issue-link" data-issue-key="LU-9206"><del>LU-9206</del></a>)</li> <li>each mirrored directory entry would contain two or more different FIDs referencing inodes on separate MDTs (for subdirectories), or the same FID (for regular files), similar to how ZFS Block Pointers can be referenced by and directly reference up to 3x different DVAs (block numbers) that have copies of the same data</li> <li>each mirrored directory inode would have the full layout of all shards in the directory, and client can determine which shard to use for lookup</li> <li>updates to the mirrored directory would always need distributed transactions that inserted or removed the redundant dirents together</li> <li>normal DNE distributed transaction recovery would apply to recover incomplete transactions if an MDT is offline during an update</li> </ul> </li> </ul> Related LU-10329 LU-7426 LU-4215 LU-7319 LU-7427 LU-9206 LU-16742 Development Rank 1|i00gf3: Rank (Obsolete) 9223372036854775807