https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9007 Lustre <p>The current MDS object allocator is designed only to allocate objects for one file at the time the file is first created. For progressive file layouts, at a minimum the allocator will need to be enhanced in order to avoid allocating objects on OSTs that are already part of a file's other components. If files have multiple objects allocated to the same OSTs before objects are allocated from unused OSTs, there may be a significant performance loss due to oversubscribing the bandwidth on that OST compared to the other OSTs. The only exception may be for a fully-striped component at the end of the file (see <a href="http://wiki.lustre.org/Progressive_File_Layouts#Example_Progressive_Layouts" class="external-link" target="_blank" rel="nofollow noopener">Example Progressive Layouts</a> for more detail), where it would be acceptable to allocate objects across all of the available OSTs to maximize the bandwidth available for the file.</p> LU-9007

Improved object allocator for FLR composite files

Improvement Minor Resolved Fixed Zhenyu Xu Joseph Gmitter FLR2 Wed, 11 Jan 2017 13:08:49 +0000 Fri, 13 May 2022 00:13:50 +0000 Thu, 9 Aug 2018 19:06:17 +0000 Lustre 2.12.0 0 6 <p>This work is also a pre-requisite for FLR-related improvements to the MDS object allocator. While PFL requires that the objects are preferably not on the same OSTs between components, this is not a hard requirement. At worst this impacts performance, and in some cases (e.g. widely striped last component) it may even be desirable to re-use the same OSTs in order to maximize the bandwidth of large files.</p> <p>FLR has similar, but more specific requirements for OST selection on components with overlapping extents, in order of decreasing priority:</p> <ol> <li>objects with overlapping components <b>must not</b> share the same OST.  Implies max replica count == OST count/component stripe count.  In theory this could be relaxed if all replicas have the same <tt>stripe_count</tt> and <tt>stripe_size</tt>, then it would only require that the same OST cannot be at the same <tt>stripe_index</tt> of different components, in which case max replica count == OST count.</li> <li>objects with overlapping components <b>should not</b> share OSTs on the same OSS node (by NID from <tt>imp-&gt;imp_connection-&gt;c_peer.nid</tt>, as <tt>qos_add_tgt()</tt> does) to avoid the shared node failure domain.</li> <li>objects with overlapping components <b>should not</b> share OSTs on the same OSS failover pair (by failover NID from <tt>imp-&gt;imp_conn_list.oic_conn-&gt;c_peer.nid</tt>, as <tt>lprocfs_import_seq_show()</tt> does) to avoid the shared storage enclosure/controller failure domain.  There may be other OSS nodes that share the same storage enclosure/controller, but there isn't any way for the client to determine this automatically.</li> <li>objects with overlapping components <b>should not</b> be on OSTs on the same network switch, power supply, rack, etc. but this depends on external information that is not currently available to Lustre. That could optionally be added via a separate configuration file/options, but the above cases will automatically cover</li> </ol> <p>Do we actually distinguish if the components are for generic PFL or FLR? it seems like to be a bad idea to me to know that information at LOD layer. I would like to make this allocation policy as generic and best-effort.</p> <p>Sparse OST index has been supported for a long time. How do you think if we partition OST indices based on the distance? The distance is defined by servers, racks, and switches. Anyway, more information the allocation policy can get the better decision it can make.</p> <p>I think encoding anything into the OST index is a non-starter. This would totally break for existing filesystems, and administrators would have a hard time getting it right, and then it would break if they needed to move nodes around for some reason. We already have OSS and OSS failover information in the LOD, so we may as well use it. In fact, the QOS RR allocator already spreads stripes across OSS nodes to avoid contention if possible. We can add in other rack/switch/power information later if we actually need it.</p> <p>I don't think that understanding "PFL" vs. "FLR" in LOD is quite the right thing, but rather it will understand layout components and whether they are sequential of overlapping, and select the best OSTs in that case.</p> <p>One simple proposal is to check the NID of each OST to put automatically separate OSTs into fault domains based on which OSS that are located on. This is not perfect, but is simple and works for all existing systems without additional input from the administrator. The existing QOS RR allocator will already prefer to distribute allocations across OSS nodes if possible, but this can be extended to actually and a requirement for all allocations. </p> <p>Secondly, a simple integer <tt>domain</tt> value can be assigned to each OST by the administrator, and the LOD can use this to separate OSTs into independent groups. OSTs with the same <tt>domain</tt> number should not be used for redundancy for other components. </p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/32404" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32404</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9007" title="Improved object allocator for FLR composite files" class="issue-link" data-issue-key="LU-9007"><del>LU-9007</del></a> lod: improve obj alloc for FLR file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1a0c093c55d034ba6013f05f7c2a68664d3d0901</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/32813" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32813</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9007" title="Improved object allocator for FLR composite files" class="issue-link" data-issue-key="LU-9007"><del>LU-9007</del></a> lod: get rid of comp ost in use array<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 73c65d7e3402e2465a0b1042eb7ccaf185730b87</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32404/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32404/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9007" title="Improved object allocator for FLR composite files" class="issue-link" data-issue-key="LU-9007"><del>LU-9007</del></a> lod: improve obj alloc for FLR file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fabf3fe7ac06d916d8c433a99f1f4a4bd3632638</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32813/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32813/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9007" title="Improved object allocator for FLR composite files" class="issue-link" data-issue-key="LU-9007"><del>LU-9007</del></a> lod: get rid of comp ost in use array<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a277952c65d4aad1abb9ac9f759af16a43902068</p> <p>Landed for 2.12</p> Related LU-8998 LU-10158 LU-15834 LU-15841 LU-11238 Development Rank 1|hzz0dz: Rank (Obsolete) 9223372036854775807 Severity