https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12757 Lustre <p>We see sanity-lfsck test_36a fail in resync for the last two of the following three calls to ‘lfs mirror resync’ from test 36a:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 5271 $LFS mirror resync $DIR/$tdir/f0 || 5272 error <span class="code-quote">"(6) Fail to resync $DIR/$tdir/f0"</span> 5273 $LFS mirror resync $DIR/$tdir/f1 || 5274 error <span class="code-quote">"(7) Fail to resync $DIR/$tdir/f1"</span> 5275 $LFS mirror resync $DIR/$tdir/f2 || 5276 error <span class="code-quote">"(8) Fail to resync $DIR/$tdir/f2"</span> 5277 </pre> </div></div> <p>It looks like this test started failing with these two errors on 07-September-2019 with Lustre master version 2.12.57.54.</p> <p>Looking at the suite_log for <a href="https://testing.whamcloud.com/test_sets/a5f2b938-d438-11e9-a2b6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a5f2b938-d438-11e9-a2b6-52540065bddc</a>, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs mirror mirror: component 131075 not synced : No space left on device (28) lfs mirror mirror: component 131076 not synced : No space left on device (28) lfs mirror mirror: component 196613 not synced : No space left on device (28) lfs mirror: '/mnt/lustre/d36a.sanity-lfsck/f1' llapi_mirror_resync_many: No space left on device. sanity-lfsck test_36a: @@@@@@ FAIL: (7) Fail to resync /mnt/lustre/d36a.sanity-lfsck/f1 </pre> </div></div> <p>Similarly, looking at the suite_log for <a href="https://testing.whamcloud.com/test_sets/42fbb9fe-d575-11e9-9fc9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/42fbb9fe-d575-11e9-9fc9-52540065bddc</a>, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs mirror mirror: component 131075 not synced : No space left on device (28) lfs mirror mirror: component 131076 not synced : No space left on device (28) lfs mirror mirror: component 196613 not synced : No space left on device (28) lfs mirror: '/mnt/lustre/d36a.sanity-lfsck/f2' llapi_mirror_resync_many: No space left on device. sanity-lfsck test_36a: @@@@@@ FAIL: (8) Fail to resync /mnt/lustre/d36a.sanity-lfsck/f2 </pre> </div></div> <p>It is possible that we are running out of disk space on an OST, but it seems strange that this just started earlier this month.</p> <p>Logs for other failures are at<br/> <a href="https://testing.whamcloud.com/test_sessions/279dd05c-e122-4f8f-bafe-b8299e8e0e61" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/279dd05c-e122-4f8f-bafe-b8299e8e0e61</a><br/> <a href="https://testing.whamcloud.com/test_sessions/fe936f3a-df7d-4d23-9d28-721da7ab8f76" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/fe936f3a-df7d-4d23-9d28-721da7ab8f76</a></p> LU-12757

sanity-lfsck test 36a fails with '(N) Fail to resync /mnt/lustre/d36a.sanity-lfsck/f2'

Bug Minor Resolved Fixed WC Triage James Nunez HPv2 Thu, 12 Sep 2019 20:11:56 +0000 Mon, 16 Jan 2023 22:30:45 +0000 Wed, 12 Aug 2020 19:38:19 +0000 Lustre 2.13.0 Lustre 2.12.3 Lustre 2.12.4 Lustre 2.12.5 Lustre 2.14.0 0 7 <p>I added 'lfs df' to a patch that hits this problem consistently, patch <a href="https://review.whamcloud.com/33919/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33919/</a> , and it doesn't look like we are out of space</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ======================================= 22:37:15 (1568327835) ##### The target MDT-object's LOV EA corrupted as to lose one of the mirrors information. The layout LFSCK should rebuild the LOV EA with the PFID EA of related OST-object(s) belong to the mirror. ##### 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.126361 s, 33.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0600958 s, 69.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0601316 s, 69.8 MB/s UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 43584 2168 37428 6% /mnt/lustre[MDT:0] lustre-OST0000_UUID 71100 3452 57504 6% /mnt/lustre[OST:0] lustre-OST0001_UUID 71100 1272 62396 2% /mnt/lustre[OST:1] lustre-OST0002_UUID 71100 1272 61948 3% /mnt/lustre[OST:2] lustre-OST0003_UUID 71100 1880 61748 3% /mnt/lustre[OST:3] filesystem_summary: 284400 7876 243596 4% /mnt/lustre lfs mirror mirror: component 131075 not synced : No space left on device (28) lfs mirror mirror: component 131076 not synced : No space left on device (28) lfs mirror mirror: component 196613 not synced : No space left on device (28) lfs mirror: '/mnt/lustre/d36a.sanity-lfsck/f2' llapi_mirror_resync_many: No space left on device. sanity-lfsck test_36a: @@@@@@ FAIL: (8) Fail to resync /mnt/lustre/d36a.sanity-lfsck/f2 </pre> </div></div> <p>It is very likely that the culprit is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 0f670d1ca9dd5af697bfbf3b95a301c61a8b4447 Author: Bobi Jam &lt;bobijam@whamcloud.com&gt; AuthorDate: Wed Oct 10 14:23:55 2018 +0800 LU-11239 lfs: fix mirror resync error handling This patch returns error for partially successful mirror resync. Signed-off-by: Bobi Jam &lt;bobijam@whamcloud.com&gt; Change-Id: I9d6c9ef5aca1674ceb7a9cbc6b790f3f7276ff5d Reviewed-on: https://review.whamcloud.com/33537 </pre> </div></div> <p>though this is just <em>returning</em> the error, it isn't <em>causing</em> the error, AFAICS.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36176" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36176</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12757" title="sanity-lfsck test 36a fails with &#39;(N) Fail to resync /mnt/lustre/d36a.sanity-lfsck/f2&#39;" class="issue-link" data-issue-key="LU-12757"><del>LU-12757</del></a> utils: avoid newline inside error message<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4d39bed4489f3b38b388ec69e449fdc65afe6f19</p> <p>The above patch does not fix the test failure here, it is just cosmetic to fix the error message to not have a newline in the middle:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs mirror mirror: component 131075 not synced : No space left on device (28) </pre> </div></div> <p>I also notice that <tt>progname</tt> is "<tt>lfs mirror mirror</tt>", which is also not correct. That is because one or more of <tt>lfs_setstripe_internal()</tt> is appending <tt>argv<span class="error">&#91;0&#93;</span></tt> to <tt>progname</tt> internally (via the <tt>cmd[]</tt> buffer), <b>and</b> printing both <tt>progname</tt> and <tt>argv<span class="error">&#91;0&#93;</span></tt> explicitly in error messages, <b>and</b> <tt>lfs_mirror()</tt> is appending <tt>argv<span class="error">&#91;0&#93;</span></tt> to progname, <b>and</b> <tt>llapi_error()-&gt;error_callback_default()</tt> is appending <tt>liblustreapi_cmd</tt> as well. That is very confusing. That should be fixed in a separate patch.</p> <p>Again detected under -&gt; <a href="https://testing.whamcloud.com/test_sets/b004f920-e795-11e9-b62b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b004f920-e795-11e9-b62b-52540065bddc</a></p> <p>Found similar error on PPC client 2.12.3<br/> <a href="https://testing.whamcloud.com/test_sets/d9ac1a0c-eb0e-11e9-b62b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d9ac1a0c-eb0e-11e9-b62b-52540065bddc</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ======================================= 22:20:32 (1570573232) ##### The target MDT-object's LOV EA corrupted as to lose one of the mirrors information. The layout LFSCK should rebuild the LOV EA with the PFID EA of related OST-object(s) belong to the mirror. ##### 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.147001 s, 28.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0381554 s, 110 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0381345 s, 110 MB/s lfs mirror mirror: cannot get WRITE lease, ext 1: Device or resource busy (16) lfs mirror: '/mnt/lustre/d36a.sanity-lfsck/f0' llapi_lease_get_ext resync failed: Device or resource busy. sanity-lfsck test_36a: @@@@@@ FAIL: (6) Fail to resync /mnt/lustre/d36a.sanity-lfsck/f0 </pre> </div></div> <p>It looks like the small filesystems used by sanity-lfsck.sh are causing problems in this test, because the client is consuming all of the free space as grant because "<tt>lfs mirror resync</tt>" is using <tt>O_DIRECT</tt> to do the writes, which currently does not consume grants.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36176/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36176/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12757" title="sanity-lfsck test 36a fails with &#39;(N) Fail to resync /mnt/lustre/d36a.sanity-lfsck/f2&#39;" class="issue-link" data-issue-key="LU-12757"><del>LU-12757</del></a> utils: avoid newline inside error message<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3f3a241498be7e043df7e416da7fc8722a559498</p> <p>Landed for 2.14</p> <p>Reopening this ticket because patch that landed changes error message formatting but does not address the problem described in this ticket. </p> <p>To fix this, we need one of the patches from <a href="https://jira.whamcloud.com/browse/LU-4664" title="sync write should consume grant on client" class="issue-link" data-issue-key="LU-4664"><del>LU-4664</del></a> and/or <a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> to be landed, so that <tt>O_DIRECT</tt> writes used by resync do not consume all of the grants.</p> <p>+1 on master:<br/> <a href="https://testing.whamcloud.com/test_sets/18633cde-dc95-4d48-add3-405591582c3f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/18633cde-dc95-4d48-add3-405591582c3f</a></p> <p>Patch <a href="https://review.whamcloud.com/35896" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35896</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O</tt>" is in master-next and should resolve this issue. This ticket can be closed once that patch lands and this problem is no longer seen.</p> <p>Additional details about test, I hope the will be useful to reproduce issue.  In most cases failed tests contains only single MDT (list from link below was analyzed for sanity-lfsck.sh/test36a) </p> <p><a href="https://testing.whamcloud.com/search?horizon=2332800&amp;test_set_script_id=4f25830c-64fe-11e2-bfb2-52540035b04c&amp;sub_test_script_id=1bd8f58e-6f10-11e8-a55d-52540065bddc&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/search?horizon=2332800&amp;test_set_script_id=4f25830c-64fe-11e2-bfb2-52540035b04c&amp;sub_test_script_id=1bd8f58e-6f10-11e8-a55d-52540065bddc&amp;source=sub_tests#redirect</a></p> <p>However, sometimes tests with one MDT passed as well.</p> <p> </p> <p>For example failed tests: output from test</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 43584 2444 37152 7% /mnt/lustre[MDT:0] lustre-OST0000_UUID 71100 7556 56544 12% /mnt/lustre[OST:0] lustre-OST0001_UUID 71100 5376 58724 9% /mnt/lustre[OST:1] lustre-OST0002_UUID 71100 10496 50508 18% /mnt/lustre[OST:2] lustre-OST0003_UUID 71100 1280 61948 3% /mnt/lustre[OST:3]</pre> </div></div> <p>Test where multiple MDTs used - passed: output from test</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 283520 3968 277504 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 283520 3200 278272 2% /mnt/lustre[MDT:1] lustre-MDT0002_UUID 283520 3200 278272 2% /mnt/lustre[MDT:2] lustre-MDT0003_UUID 283520 3200 278272 2% /mnt/lustre[MDT:3] lustre-OST0000_UUID 282624 16384 264192 6% /mnt/lustre[OST:0] lustre-OST0001_UUID 282624 10240 270336 4% /mnt/lustre[OST:1] lustre-OST0002_UUID 282624 22528 258048 9% /mnt/lustre[OST:2] lustre-OST0003_UUID 282624 4096 276480 2% /mnt/lustre[OST:3] </pre> </div></div> <p>When OST's number less than 3 test is skipped, so in this cases no failure in test.</p> <p> </p> <p> </p> <p> </p> <p> </p> <p>The patch that Andreas references that should fix this issue, <a href="https://review.whamcloud.com/#/c/35896/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35896/</a>, landed to master on July 10 and we haven’t seen this issue since July 9. It looks like this issue is fixed and we can close this ticket.</p> Related LU-4664 LU-11239 LU-12687 Development Rank 1|i00mpb: Rank (Obsolete) 9223372036854775807 Severity