https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11760 Lustre <p>We faced multiple missing OST objects mdtest job failure during failover/failback test(stripe count 2).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>V-1: Entering unique_dir_access... V-1: Entering mdtest_stat... 08/14/2018 21:46:25: Process 25(nid00016): FAILED in mdtest_stat, unable to stat file: No such file or directory 08/14/2018 21:46:25: Process 30(nid00045): FAILED in mdtest_stat, unable to stat file: No such file or directory</pre> </div></div> <p>Failure occurred because of absent the range of objects on one of the OSTs.</p> <p>The marker of the failure could be the following message after recovery on OST:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 14 21:46:07 snx11205n004 kernel: format at ofd_dev.c:1713:ofd_create_hdl doesn't end in newline </pre> </div></div> LU-11760

formatted OST recognition change

Bug Critical Resolved Fixed Sergey Cheremencev Sergey Cheremencev patch Tue, 11 Dec 2018 14:24:15 +0000 Thu, 12 Sep 2019 04:18:15 +0000 Wed, 21 Aug 2019 11:26:27 +0000 Lustre 2.13.0 Lustre 2.12.3 0 2 <p>The reason of failures is uncommitted journal transaction.<br/> By default jbd2 commits transaction each 5 seconds or in case when journal buffers are full.<br/> Create of one object doesn't need a lot of space(inode size + direntry). Thus usually under mdtest load<br/> journal buffers don't become full and commit is occurred every 5 seconds.<br/> In respect that OST creates objects asynchronously(except several rare cases), we can get a situation<br/> when MDS think that successfully created 100 000 objects despite these objects are not written to disk(OST side) .<br/> In case of failover these uncommitted objects could be lost and should be recreated after establishing connection from OST to MDT.<br/> MDT sends LAST_ID to OST and it should recreate missing object.<br/> This scheme works fine for a long time however there is exclusion for the case when the difference between MDS_Last_ID and OST_Last_ID is bigger then 100 000 (5 * OST_MAX_PRECREATE).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* This can happen if a new OST is formatted and installed * in place of an old one at the same index. Instead of * precreating potentially millions of deleted old objects * (possibly filling the OST), only precreate the last batch. * LFSCK will eventually clean up any orphans. LU-14 */ if (diff &gt; 5 * OST_MAX_PRECREATE) { diff = OST_MAX_PRECREATE / 2; LCONSOLE_WARN("%s: Too many FIDs to precreate " "OST replaced or reformatted: " "LFSCK will clean up", ofd_name(ofd)); </pre> </div></div> <p>In such case It decides OST is formatted and recreates only last 10 000. For us it means we can have huge gaps in OST objects.<br/> I think 100 000 constant was actual for old systems and should be revised.</p> <p>Sergey Cheremencev (c17829@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/33833" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33833</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a> ofd: formatted OST recognition change<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 829732b1fd8f11e4a4aa5bc7ef6509c4aa771dc3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33833/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33833/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a> ofd: formatted OST recognition change<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d07d9c5ed0aa1d6614944c7d1e0ca55cba301dc4</p> <p>Landed for 2.13</p> <p>I think that this is not a full solution to the problem, and is causing a lot of test failures for conf-sanity test_69 for OSTs that do not have 500k free inodes.</p> <p>I think there are two problems with the patch that was landed:</p> <ul class="alternate" type="square"> <li>this essentially is changing <tt>OST_MAX_PRECREATE</tt> on the OST side, but not on the MDS, so isn't properly handling the recovery</li> <li>the root of the problem is not the number of objects created <em>per RPC</em> but rather the number of objects created <em>per commit</em></li> </ul> <p>I don't mind to have a change that is increasing the maximum number of objects created per commit, but this needs to be negotiated between the MDS and OSS at connect time, with a new <tt>OBD_CONNECT2_MAX_PRECREATE</tt> flag and an <tt>ocd_max_precreate</tt> field (probably using <tt>__u32 padding1</tt>) that passes the current <tt>OST_MAX_PRECREATE</tt> value, and use <tt>OST_MAX_PRECREATE_OLD = 20000</tt> if the feature is not available. This also allows fixing the problem properly in older releases.</p> <p>Until this is fixed (and even afterward), the proper solution is to track on the OST how many objects have been created by each MDT/sequence within the current transaction (use a commit callback to reset the "created this transaction" counter to zero), and force a sync journal commit during precreate if the number exceeds <tt>OST_MAX_PRECREATE</tt>. If there are multiple MDTs creating, or the create rate is not too large, or if the clients do some IO and force a transaction commit anyway, then there is no need for a commit. This also avoids the future bug when 500k+ precreates can happen within a single commit, since we are already over 150k/s creates on the MDS, and that is processing separate RPCs while the OST is in a fast local loop precreating objects.</p> <p>I <em>was</em> going to say that the <tt>if (diff &gt; 500000)</tt> check in <tt>ofd_create_hdl()</tt> could be modified to also check if <tt>LAST_ID &lt; 5</tt> or similar, to detect if this is a newly-formatted filesystem, but I realize that this case may also happen if e.g. the OST is restored from a backup or a snapshot and has an old <tt>LAST_ID</tt> but is not new. It would be useful to update the comment to reflect this.</p> <blockquote><p>I don't mind to have a change that is increasing the maximum number of objects created per commit, but this needs to be negotiated between the MDS and OSS at connect time, with a new <tt>OBD_CONNECT2_MAX_PRECREATE</tt> flag and an <tt>ocd_max_precreate</tt> field (probably using <tt>__u32 padding1</tt>) that passes the current <tt>OST_MAX_PRECREATE</tt> value, and use <tt>OST_MAX_PRECREATE_OLD = 20000</tt> if the feature is not available. This also allows fixing the problem properly in older releases.</p></blockquote> <p>If we increase OST_MAX_PRECREATE on MDS also, I am afraid MDS will ask OST to precreate too rare. It may cause several different problems. For example OST after failover needs to precreate more objects that takes extra time(500 000 instead of 20 000).</p> <p> I suggest to use the 2nd approach with commit callback and revert <a href="https://review.whamcloud.com/33833" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33833</a>.<br/> I will push a patch.</p> <p>Sergey Cheremencev (c17829@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/35373" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35373</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a> ofd: limit num of objects to create in 1 transaction<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e75f4977695657710a9144a31a0d149fa1925ef8</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35388" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35388</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a> ofd: formatted OST recognition change"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 563277b8d728c45bd89074c07a22f1432beea344</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35388/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35388/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a> ofd: formatted OST recognition change"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8065d44c0a2b29885ca429674ccab7785d2db08b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35373/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35373/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a> ofd: limit num of objects to create in 1 transaction<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4485ee8be4cf224e2543f6344efc6e1cb295a0a7</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35951" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35951</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a> ofd: limit num of objects to create in 1 transaction<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: e08f46ed1e8fc4daf8ac092ca305438ea354ec24</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35951/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35951/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a> ofd: limit num of objects to create in 1 transaction<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 963559b3087bcbb0bdd541c983085eff7feca882</p> Related LU-12415 LU-12404 Development Rank 1|i007s7: Rank (Obsolete) 9223372036854775807 Severity