https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12273 Lustre <p><a href="https://jira.whamcloud.com/browse/LU-9846" title="Overstriping - more than stripe per OST per component" class="issue-link" data-issue-key="LU-9846"><del>LU-9846</del></a> describes the overstriping feature for data, which allows placing &gt; 1 stripe per OST.  This can be done with MDTs as well, for somewhat similar reasons:</p> <p>"it allows more concurrency on the MDT, exceeding single-directory size limitations, directory migration/compaction, etc." (per Andreas)</p> <p>This exists in limited form today, accessible with a fail loc:<br/> OBD_FAIL_LARGE_STRIPE (0x1703)</p> <p>Which is used in sanity test 300k to put a bunch of stripes on MDT0:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> #define OBD_FAIL_LARGE_STRIPE 0x1703 $LCTL set_param fail_loc=0x1703 $LFS setdirstripe -i 0 -c192 $DIR/$tdir/striped_dir || error "set striped dir err </pre> </div></div> <p>Actually doing this as a feature requires various other enabling changes, but this test shows it should be possible.  It's also possible to use the method in this test to create temporary setups for benchmarking this idea to confirm it's worth pursuing.</p> LU-12273

DNE3: Metadata overstriping

New Feature Minor Open Unresolved Patrick Farrell Patrick Farrell dne3 Wed, 8 May 2019 18:17:27 +0000 Mon, 22 Jan 2024 15:49:53 +0000 0 9 <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35034" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35034</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12273" title="DNE3: Metadata overstriping" class="issue-link" data-issue-key="LU-12273">LU-12273</a> lod: metadata overstriping<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c3bfede6854263915c967f53602d0a1e46d6d1bf</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35939" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35939</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12273" title="DNE3: Metadata overstriping" class="issue-link" data-issue-key="LU-12273">LU-12273</a> lod: Trivial metadata overstriping<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6e97b13d5dd0ba3d49b5c5191f4733e95663e573</p> <blockquote> <p>if we compare 2 x MDT per MDS vs MDT Overstriping=2, it's 2 x jbd2 thread vs still single journal thread, etc?</p></blockquote> <p>Correct - with overstriping there is still only a single journal/device and some filesystem locks, while two separate MDTs have totally separate infrastructure (but each one is 1/2 the size and needs more space balancing, double journal memory usage). If we can get "close" performance with 2x or 4x overstriping vs. 2x or 4x MDTs then using directory overstriping would be better overall.</p> <p>"Patrick Farrell &lt;farr0186@gmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49707" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49707</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12273" title="DNE3: Metadata overstriping" class="issue-link" data-issue-key="LU-12273">LU-12273</a> obd: Reserve metadata overstriping flags<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 91f41501f508166e33881d00ac795f9296f8978c</p> <p>So this is fascinating.</p> <p>Each MDT does not know about itself in the pool code, because it's the local device, so it's handled differently.<br/> The list of targets (in the QOS/RR pool code) on an MDT is <b>the other MDTs in the system</b>.</p> <p>The practical result of this is the first MDT is not selected by the allocation code <b>on the MDT</b>, so it only<br/> gets one stripe on it.</p> <p>eg, with 2 MDTs:</p> <p>lmv_stripe_count: 8 lmv_stripe_offset: 1 lmv_hash_type: crush,overstriped<br/> mdtidx           FID<span class="error">&#91;seq:oid:ver&#93;</span><br/>      1           <span class="error">&#91;0x240000400:0x2:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000401:0x2:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000401:0x3:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000401:0x4:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000401:0x5:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000401:0x6:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000401:0x7:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000401:0x8:0x0&#93;</span></p> <p>Or, with 4 MDTs, it can look like this: <br/> lmv_stripe_count: 16 lmv_stripe_offset: 3 lmv_hash_type: crush,overstriped<br/> mdtidx           FID<span class="error">&#91;seq:oid:ver&#93;</span><br/>      3           <span class="error">&#91;0x2c0000400:0x6:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000403:0x10:0x0&#93;</span><br/>      1           <span class="error">&#91;0x240000402:0x11:0x0&#93;</span><br/>      2           <span class="error">&#91;0x280000401:0x11:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000403:0x11:0x0&#93;</span><br/>      1           <span class="error">&#91;0x240000402:0x12:0x0&#93;</span><br/>      2           <span class="error">&#91;0x280000401:0x12:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000403:0x12:0x0&#93;</span><br/>      1           <span class="error">&#91;0x240000402:0x13:0x0&#93;</span><br/>      2           <span class="error">&#91;0x280000401:0x13:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000403:0x13:0x0&#93;</span><br/>      1           <span class="error">&#91;0x240000402:0x14:0x0&#93;</span><br/>      2           <span class="error">&#91;0x280000401:0x14:0x0&#93;</span><br/>      0           <span class="error">&#91;0x200000403:0x14:0x0&#93;</span><br/>      1           <span class="error">&#91;0x240000402:0x15:0x0&#93;</span><br/>      2           <span class="error">&#91;0x280000401:0x15:0x0&#93;</span></p> <p>Notice 3 is only used once.</p> <p>Allocation of the first stripe is handled like this, without reference to the pool:<br/>         /* Allocate the first stripe locally */<br/>         rc = dt_fid_alloc(env, lod-&gt;lod_child, &amp;fid, NULL, NULL);<br/>         if (rc &lt; 0)<br/>                 GOTO(out, rc);</p> <p>        stripes<span class="error">&#91;0&#93;</span> = dt_locate_at(env, lod-&gt;lod_child, &amp;fid,<br/>                                   dt-&gt;do_lu.lo_dev-&gt;ld_site-&gt;ls_top_dev, &amp;conf);</p> <p><b>then</b> the qos/rr alloc code is called to allocate the rest of the stripes.</p> <p>I'm not sure what to do about this - The device init process doesn't really seem something to mess with.<br/> I'm thinking the right thing to do is special case this for RR + overstriping.</p> <p>Basically, add one more to the range of indices that can be selected during RR, and if it's found, then do<br/> a local allocation.  It complicates the code slightly but it's the only solution that seems sane.</p> <p>I'll do that if there's not an objection.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49707/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49707/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12273" title="DNE3: Metadata overstriping" class="issue-link" data-issue-key="LU-12273">LU-12273</a> obd: Reserve metadata overstriping flags<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8b62a8ca9c2e824d94fbf8bc266b4028e9b5fa63</p> Related LU-9846 LU-12926 Development Rank 1|i00fz3: Rank (Obsolete) 9223372036854775807