https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12040 Lustre <p>In 2011y, Johann was introduce a wire protocol changes.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit f90abfdc961debae069804307dcbc883b50c137c Author: Johann Lombardi &lt;johann@whamcloud.com&gt; Date: Thu Dec 15 01:00:00 2011 +0100 LU-169 lov: add generation number to LOV EA </pre> </div></div> <p>This commit replace an unused field 'stripe_offset' in server reply with layout generation.<br/> so same offset in this structure have a different purpose.<br/> But, author forget about replay case. in this case, LOV EA directly copied from Server reply into client source buffer. Before this change it was LOV_DEFAULT_OFFSET aka -1, but after this change it's replaced with zero for new allocated file.<br/> PFL landing have improve a layout checks for 'SETSTRIPE' requests, so it verified against a pool indexes now.<br/> &#8211; <br/> That's all. </p> <p>Client create a file in directory with pool assigned, but server failed. Client tries a resend open+create call but it have silence failed on replay with EINVAL in lod_verify_v1v3 as '0' isn't part of lod pool indexes.</p> LU-12040

File lost during recovery

Bug Critical Resolved Fixed Alexey Lyashkov Alexey Lyashkov patch Mon, 4 Mar 2019 08:02:18 +0000 Tue, 21 Jul 2020 11:15:59 +0000 Tue, 10 Sep 2019 17:38:22 +0000 Lustre 2.10.0 Lustre 2.11.0 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.13.0 0 7 <p>Vladimir Saveliev (c17830@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/34369" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34369</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12040" title="File lost during recovery" class="issue-link" data-issue-key="LU-12040"><del>LU-12040</del></a> tests: test replay for creation of pooled file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8d2cf3d0b8265eda1fa550b2fdfdba4d6a2b2ad7</p> <p>Vladimir Saveliev (c17830@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/34371" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34371</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12040" title="File lost during recovery" class="issue-link" data-issue-key="LU-12040"><del>LU-12040</del></a> mdc: reset lmm-&gt;lmm_stripe_offset in mdc_save_lovea<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3f9442ac7685d7e017269edb1c80c1938c05a46b</p> <p><a href="https://review.whamcloud.com/#/c/34370/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34370/</a> and  <a href="https://review.whamcloud.com/34371" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34371</a> are two possible solutions for the problem.  <a href="https://review.whamcloud.com/34369" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34369</a> is a test illustrating the issue.</p> <p>I would say that patch:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 89693927f0b065d44fdc496f6b49539118570104 LU-8998 lod: accomodate to composite layout Modify the LOD to make it support the composite layout: : : - Object allocation code is adjusted to not only check the used OSTs in this round of allocation, but also the used OSTs in the existing layout components.. Reviewed-on: https://review.whamcloud.com/24823 </pre> </div></div> <p>is what triggered this to be a problem in newer releases, which was landed as v2_9_55_0-14-g8969392. </p> <p>Is <a href="https://review.whamcloud.com/34371" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34371</a> ready to land? It has Code-Review +1 from Andreas Dilger and Mike Pershin, and Verified +1 from Jenkins and Maloo.</p> <p>This patch and the prerequisite patch are in the master-next branch and should probably land by next week, depending on how integration testing goes. </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34371/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34371/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12040" title="File lost during recovery" class="issue-link" data-issue-key="LU-12040"><del>LU-12040</del></a> mdc: reset lmm-&gt;lmm_stripe_offset in mdc_save_lovea<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c872afa36ff5de5910f0f524f7a487982fa0c776</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34919" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34919</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12040" title="File lost during recovery" class="issue-link" data-issue-key="LU-12040"><del>LU-12040</del></a> mdc: reset lmm-&gt;lmm_stripe_offset in mdc_save_lovea<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 497c0fad736c9365e83b6ee3f07b52571fe70c6c</p> <p>We're seeing continuous failures on replay-single test_134 when this patch is backported to b2_12. Shadow, do you know if there are any other patches that this one depends on to work?</p> <p>Andreas,</p> <p>I think it something wrong with test env<br/> &gt;&gt;<br/> [ 8982.834142] LustreError: 17948:0:(lod_qos.c:1358:lod_alloc_specific()) Start index 0 not found in pool 'pool_134'<br/> &gt;&gt;<br/> I have no idea why touch start to allocate a file with don't existent index in pool, but it unlikely related to this patch.</p> <p>That error with stripe offset actually seems <b>really</b> likely to be related to this patch, which is changing how the stripe offset is handled...?</p> <p>Patrik,</p> <p>I'm sorry, but this patch do nothing with "how stripe offset is handled" in create path. This patch affects a replay code path,<br/> but if you open an console logs, you can see this error was returned <em>before</em> recovery.<br/> One operation do a object creation before recovery is 'touch' which do nothing with setstripe operation and inherit an create info from pool assigned to the parent dir.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 8981.540879] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8981.873832] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8982.658245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8982.825716] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8982.834142] LustreError: 17948:0:(lod_qos.c:1358:lod_alloc_specific()) Start index 0 not found in pool <span class="code-quote">'pool_134'</span> [ 8983.012511] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1<span class="code-quote">' '</span> /proc/mounts || <span class="code-keyword">true</span> [ 8983.340608] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8986.891724] LustreError: 137-5: lustre-MDT0000_UUID: not available <span class="code-keyword">for</span> connect from 10.2.8.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> + + mkdir $DIR/$tdir + $LFS setstripe -p pool_134 $DIR/$tdir + + replay_barrier mds1 + + touch $DIR/$tdir/$tfile &lt;&lt;&lt; create + + fail mds1 </pre> </div></div> <p>As I say before, if someone can replicate this with full debug info and attach this into ticket I can help with understanding a problem, but my view it's something bad with creating object with pool.</p> <p>This is fixed on master so this ticket should be marked RESOLVED. We should track any efforts to address this issue on 2.12.x separately.</p> Related LU-13809 Development Rank 1|i00cnb: Rank (Obsolete) 9223372036854775807 Severity