https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11239 Lustre <p>sanity-lfsck test_36a started failing on August 6, 2018 with Lustre version 2.11.53.52 build #3774. Note, sanity-lfsck test 36a landed to master with build #3774. </p> <p>In the test_log for these failures, for example at <a href="https://testing.whamcloud.com/test_sets/0e30317a-9ad2-11e8-a9f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0e30317a-9ad2-11e8-a9f7-52540065bddc</a>, we see the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>… /mnt/lustre/d36a.sanity-lfsck/f0 lcm_layout_gen: 10 lcm_mirror_count: 3 lcm_entry_count: 6 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x9b6:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x768:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 2 lmm_objects: - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x78e:0x0] } lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x769:0x0] } - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x788:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x9bc:0x0] } lcme_id: 196613 lcme_mirror_id: 3 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 3145728 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 2 lmm_objects: - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x789:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x100000000:0x9b7:0x0] } lcme_id: 196614 lcme_mirror_id: 3 lcme_flags: init,stale lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x76e:0x0] } … Inject failure, to simulate the case of missing one mirror in LOV CMD: trevis-7vm4 /usr/sbin/lctl set_param fail_loc=0x1616 fail_loc=0x1616 error: lfs mirror split: setting 'stale' is not supported sanity-lfsck test_36a: @@@@@@ FAIL: (12) Fail to split 1st mirror from /mnt/lustre/d36a.sanity-lfsck/f0 </pre> </div></div> <p>In comparing sanity-lfsck test 36a that pass testing and those that don’t, the ones that fail have components with “init,stale” flags and the tests that pass don’t have “stale flags</p> <p>More logs for this failure are at<br/> <a href="https://testing.whamcloud.com/test_sets/1131f45c-99d4-11e8-a9f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1131f45c-99d4-11e8-a9f7-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/6481868c-9ab4-11e8-a9f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6481868c-9ab4-11e8-a9f7-52540065bddc</a></p> LU-11239

sanity-lfsck test 36a fails with 'Fail to split mirror'

Bug Major Resolved Fixed Zhenyu Xu James Nunez Mon, 13 Aug 2018 19:19:31 +0000 Thu, 21 Nov 2019 21:33:56 +0000 Mon, 9 Sep 2019 18:06:20 +0000 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.12.1 Lustre 2.12.2 Lustre 2.12.3 Lustre 2.13.0 Lustre 2.12.4 0 9 <p>Lai</p> <p>This test was added with <a href="https://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=36ba989752c62cc76b06089373fcd6cec6da9008" class="external-link" target="_blank" rel="nofollow noopener">https://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=36ba989752c62cc76b06089373fcd6cec6da9008</a> . Should we revert this change or is it just a faulty test that we should add to the ALWAYS_EXCEPT list until it is corrected?</p> <p>Peter</p> <p>IMO it's test script problem. Bobi is more familiar with related code, maybe he knows more.</p> <p>Bobi</p> <p>Any suggestions?</p> <p>Peter</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/33330" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33330</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11239" title="sanity-lfsck test 36a fails with &#39;Fail to split mirror&#39;" class="issue-link" data-issue-key="LU-11239"><del>LU-11239</del></a> test: sanity-lfsck test_36a fix<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c19a05a5c896ced8dfe962df409cb0892f9e3950</p> <p>Can you please explain why we need to call <tt>lfs mirror resync</tt> multiple times on a file to make it correct? If this is a problem for regular users, then it could lead to data loss if they <em>think</em> the file is sync'd to all of the mirrors (e.g. changelog watcher that is calling "<tt>lfs mirror resync</tt>" on each file once), but in fact there are mirrors that are not uptodate.</p> <p>If there is a bug in how resync is working, then that should be fixed in lfs, and not in the test script. If this is really a bug in the test script, please explain why, so that we are sure that there is not going to be data loss for the users (and we can land the existing patch and get it out of the way for 2.12).</p> <p>Don't know why this case why resync does not successfully sync some components. And lfs mirror resync was designed to not report failure if some components are not sync-ed for some reason, but I don't see why the test_36a() could fail the resync.</p> <p><a href="https://testing.whamcloud.com/test_logs/85810774-d76b-11e8-82f2-52540065bddc/download" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_logs/85810774-d76b-11e8-82f2-52540065bddc/download</a> (client log)<br/> shows that -28 is from write</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000008:00000001:1.0:1540371611.978254:0:13766:0:(osc_request.c:1770:osc_brw_fini_request()) Process leaving via out (rc=18446744073709551588 : -28 : 0xffffffffffffffe4) 00000008:00000001:1.0:1540371611.978256:0:13766:0:(osc_request.c:1905:osc_brw_fini_request()) Process leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4) 00000008:00000002:1.0:1540371611.978257:0:13766:0:(osc_request.c:2025:brw_interpret()) request ffff893865d41800 aa ffff893865d41970 rc -28 </pre> </div></div> <p>And the OST log<br/> <a href="https://testing.whamcloud.com/test_logs/8580f360-d76b-11e8-82f2-52540065bddc/download" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_logs/8580f360-d76b-11e8-82f2-52540065bddc/download</a> (ost log)<br/> shows</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000001:00000001:0.0:1540371612.134157:0:12237:0:(osd_io.c:1810:osd_declare_write()) Process leaving (rc=0 : 0 : 0) 00080000:00000001:0.0:1540371612.134160:0:12237:0:(osd_handler.c:1916:osd_trans_start()) Process leaving (rc=0 : 0 : 0) 00000001:00000002:0.0:1540371612.134161:0:12237:0:(osd_io.c:1415:osd_write_commit()) Skipping [0] == -28 00000001:00000002:0.0:1540371612.134164:0:12237:0:(osd_io.c:1415:osd_write_commit()) Skipping [1] == -28 ... 00000001:00000002:0.0:1540371612.134355:0:12237:0:(osd_io.c:1415:osd_write_commit()) Skipping [254] == -28 00000001:00000002:0.0:1540371612.134356:0:12237:0:(osd_io.c:1415:osd_write_commit()) Skipping [255] == -28 00000001:00000001:0.0:1540371612.134357:0:12237:0:(osd_io.c:330:osd_do_bio()) Process entered 00000001:00000001:0.0:1540371612.134358:0:12237:0:(osd_io.c:462:osd_do_bio()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:0.0:1540371612.134359:0:12237:0:(osd_io.c:1484:osd_write_commit()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>Running out of space is an acceptable reason to fail resync, so long as some error is printed for the user.</p> <p>However, this shouldn't happen during this particular test. Is the test file too large, or is there some leak of space?</p> <p>It has something about grant. OST reports that it has 62,066,688 bytes free, and 54,898,688 bytes available, minus the space already granted to the client, it has 897,024 left for the incoming write, and it's not enough for the requested write size (1,073,152)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00002000:00000020:0.0:1540371612.048313:0:12237:0:(tgt_grant.c:413:tgt_grant_statfs()) lustre-OST0002: cli 31207193-e3e7-03d0-da8d-e086caad4da3/ffff9f173946e000 free: 62066688 avail: 54898688 00002000:00000020:0.0:1540371612.048315:0:12237:0:(tgt_grant.c:477:tgt_grant_space_left()) lustre-OST0002: cli 31207193-e3e7-03d0-da8d-e086caad4da3/ffff9f173946e000 avail 54898688 left 897024 unstable 0 tot_grant 53999552 pending 0 00002000:00000001:0.0:1540371612.048316:0:12237:0:(tgt_grant.c:479:tgt_grant_space_left()) Process leaving (rc=897024 : 897024 : db000) 00002000:00000001:0.0:1540371612.048317:0:12237:0:(tgt_grant.c:504:tgt_grant_incoming()) Process entered 00002000:00000020:0.0:1540371612.048318:0:12237:0:(tgt_grant.c:520:tgt_grant_incoming()) lustre-OST0002: cli 31207193-e3e7-03d0-da8d-e086caad4da3/ffff9f173946e000 reports grant 43601920 dropped 0, local 46772224 00002000:00000001:0.0:1540371612.048319:0:12237:0:(tgt_grant.c:567:tgt_grant_incoming()) Process leaving 00002000:00000001:0.0:1540371612.048320:0:12237:0:(tgt_grant.c:705:tgt_grant_check()) Process entered 00002000:00000020:0.0:1540371612.048320:0:12237:0:(tgt_grant.c:812:tgt_grant_check()) lustre-OST0002: cli 31207193-e3e7-03d0-da8d-e086caad4da3/ffff9f173946e000 idx 0 no space for 1073152 00002000:00000020:0.0:1540371612.048322:0:12237:0:(tgt_grant.c:832:tgt_grant_check()) lustre-OST0002: cli 31207193-e3e7-03d0-da8d-e086caad4da3/ffff9f173946e000 granted: 0 ungranted: 0 grant: 46772224 dirty: 3170304 00002000:00000001:0.0:1540371612.048323:0:12237:0:(tgt_grant.c:855:tgt_grant_check()) Process leaving </pre> </div></div> <p>Is that because the mirror resync writes do not use the grant already held by the client? It doesn't make sense that this would fail on a test system, because there are only 2 clients that might even be holding grant.</p> <p>mirror resync uses direct IO which I think don't consider grant on the client, and OST find the (avail - grants_assigned_to_the_client) &lt; request_direct_write_size. </p> <p>I know we've gone back and forth on <tt>O_DIRECT</tt> writes consuming grant in the past, but IMHO we <em>should</em> consume grant from the client <b>if it is available</b>, and return grant from the server if available. However, we shouldn't <em>need</em> grant for <tt>O_DIRECT</tt> writes if there is none on the client.</p> <p>This would allow the best of both worlds - clients that have grant would consume it during <tt>O_DIRECT</tt> writes so that they do not run out of space on the OST. Even if space is becoming short on an OST and grant is restricted then the client can still submit large O_DIRECT writes without grant if there is any free space on the OST.</p> <p>As a starting point, we should add error messages to "<tt>lfs mirror resync</tt>" to print the <tt>-ENOSPC</tt> errors during resync, and always return an error to the shell if any resync fails (though it shouldn't <b>stop</b> resync of an error is hit, just save it until the end). We shouldn't have to dig through Lustre debug logs to find that out, and users shouldn't have to do that either. </p> <p>Jian, can you please work on fixing the <tt>lfs mirror resync</tt> error handling as described above.</p> <p>Bobijam, can you please take a look at fixing O_DIRECT writes to consume grant (if available). </p> <p>Sure, Andreas, will do.</p> <p>Yes, working on it, and revive/revise a patch (consume grant for sync write patch of <a href="https://jira.whamcloud.com/browse/LU-4664" title="sync write should consume grant on client" class="issue-link" data-issue-key="LU-4664"><del>LU-4664</del></a>) and testing whether that could work on this issue.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/33537" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33537</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11239" title="sanity-lfsck test 36a fails with &#39;Fail to split mirror&#39;" class="issue-link" data-issue-key="LU-11239"><del>LU-11239</del></a> lfs: fix mirror resync error handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d2ab5bfd9d3a5710177ea8f07c2182178dab1d2f</p> <p>it seems that the debug patch upon <a href="https://jira.whamcloud.com/browse/LU-4664" title="sync write should consume grant on client" class="issue-link" data-issue-key="LU-4664"><del>LU-4664</del></a> patch shows that the grant for sync write does fix the -ENOSPC error during resync in the sanity_lfsck test_36a case.</p> <p>+1 on Lustre b2_12 branch: <a href="https://testing.whamcloud.com/test_sets/4f773b5e-c96d-11e9-97d5-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/4f773b5e-c96d-11e9-97d5-52540065bddc</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33537/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33537/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11239" title="sanity-lfsck test 36a fails with &#39;Fail to split mirror&#39;" class="issue-link" data-issue-key="LU-11239"><del>LU-11239</del></a> lfs: fix mirror resync error handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0f670d1ca9dd5af697bfbf3b95a301c61a8b4447</p> <p>Patch landed.</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36341" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36341</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11239" title="sanity-lfsck test 36a fails with &#39;Fail to split mirror&#39;" class="issue-link" data-issue-key="LU-11239"><del>LU-11239</del></a> lfs: fix mirror resync error handling<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 3d2705121a7bedb2ebbaefae752258806d868065</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36341/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36341/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11239" title="sanity-lfsck test 36a fails with &#39;Fail to split mirror&#39;" class="issue-link" data-issue-key="LU-11239"><del>LU-11239</del></a> lfs: fix mirror resync error handling<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 195ffb9e46709c74c34cbfc2a86378ad86d2062f</p> Related LU-11111 LU-4664 LU-12757 Development Rank 1|i000mf: Rank (Obsolete) 9223372036854775807 Severity