https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10158 Lustre <p>For a mirrored file, when it is written at the first time, the policy to pick a replica as primary is primitive in current implementation, either it just chooses the first replica or a random one. A policy function should be defined for replica choosing. At least it should avoid the replicas with unavailable OSTs at this stage.</p> LU-10158

FLR: Define a replica choosing policy function

Improvement Minor Open Unresolved Zhenyu Xu Jinshan Xiong FLR2 Wed, 25 Oct 2017 00:31:44 +0000 Thu, 19 May 2022 09:08:32 +0000 0 6 <p>This should be implemented in phase 2 of FLR.</p> <p>Some thoughts on read policy, decided by the clients:</p> <ul class="alternate" type="square"> <li>any replica marked <tt>LCME_FL_STALE</tt> should be avoided</li> <li>any replica marked <tt>LCME_FL_PREF_RD</tt> should be selected first</li> <li>reads from OSTs marked <tt>OS_STATE_NONROT</tt> should be preferred (<a href="https://jira.whamcloud.com/browse/LU-14996" title="select preferred mirror using non-rotational status" class="issue-link" data-issue-key="LU-14996"><del>LU-14996</del></a>)</li> <li>reads from the local NID and/or local LNet network should be preferred</li> <li>reads from OSTs with higher performance/lower load should be preferred (<a href="https://jira.whamcloud.com/browse/LU-7880" title="add performance statistics to obd_statfs" class="issue-link" data-issue-key="LU-7880">LU-7880</a>)</li> <li>it is desirable to maximize re-use of page cache on the OST, so small files should deterministically pick the same replica (eg. first one)</li> <li>it is desirable to spread the cache for large files across OSTs to share the RAM and network bandwidth, so clients should deterministically round-robin reads for large files across replicas (e.g. every 1GB)</li> <li>if there are a large number of equivalent replicas of a file (eg. more than 3?), clients should deterministically evenly distribute their selection of the mirror by e.g. their client NID modulo mirror count (e.g. see patch <a href="https://review.whamcloud.com/29136" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29136</a> <tt>lmv_select_statfs_mdt()</tt>)</li> </ul> <p>For selection of which mirror to use for writes, decided by the MDS:</p> <ul class="alternate" type="square"> <li>any mirror marked <tt>LCME_FL_STALE</tt> must be avoided</li> <li>any replica(s) marked <tt>LCME_FL_PREF_WR</tt> should be selected first</li> <li>replicas on flash OSTs (<tt>OS_STATFS_NONROT</tt>) should be preferred</li> <li>prefer OSTs that are close network-wise to the client (same LNet network, in case of local/remote mirrors)</li> <li>writes to OSTs with higher performance/lower load should be preferred (<a href="https://jira.whamcloud.com/browse/LU-7880" title="add performance statistics to obd_statfs" class="issue-link" data-issue-key="LU-7880">LU-7880</a>)</li> <li><del>currently it is not totally safe for writes to the local NID (until we get a "no recovery sync write export" for local clients), so a remote OST should typically be used</del> (<a href="https://jira.whamcloud.com/browse/LU-12722" title="exclude local client mounted on MDS/OSS from recovery" class="issue-link" data-issue-key="LU-12722"><del>LU-12722</del></a>)</li> <li>allow specifying a "domain" number for each OST (e.g. rack, PSU, network switch), and avoid allocating replicas from the same domain. "domain" is just an arbitrary integer assigned by the admin to indicate groups of OSTs that share a common point of failure</li> </ul> <p>Converted to an issue as it is not a subtask for phase 1 of FLR (<a href="https://jira.whamcloud.com/browse/LU-9771" title="FLR1: Landing tickets for File Level Redundancy Phase 1" class="issue-link" data-issue-key="LU-9771"><del>LU-9771</del></a>).</p> Related LU-15834 LU-7880 LU-11963 LU-12722 LU-14996 LU-12649 LU-10448 LU-9007 Development Rank 1|hzzmef: Rank (Obsolete) 9223372036854775807