Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-10052] replay-single test_20b fails with 'after 4096 > before 3072' https://jira.whamcloud.com/browse/LU-10052 Lustre <p>This issue was created by maloo for wangshilong <wshilong@ddn.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/5f45e920-a61c-11e7-bb19-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5f45e920-a61c-11e7-bb19-5254006e85c2</a>.</p> <p>The sub-test test_20b failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>after 4096 > before 3072 </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: replay-single 20b</p> LU-10052

replay-single test_20b fails with 'after 4096 > before 3072'

Bug Major Resolved Fixed Hongchao Zhang Maloo zfs Sat, 30 Sep 2017 23:30:18 +0000 Thu, 29 Mar 2018 15:31:38 +0000 Thu, 25 Jan 2018 03:38:06 +0000 Lustre 2.11.0 Lustre 2.11.0 Lustre 2.10.4 0 15 <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/246f6226-b2b6-11e7-9eeb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/246f6226-b2b6-11e7-9eeb-5254006e85c2</a></p> <p>Hongchao</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>For ldiskfs, the error only occurred 3 times since Oct 1, 2016, all are related to <a href="https://review.whamcloud.com/#/c/28847/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28847/</a> for <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a>.</p> <p>It happened on ZFS many times recently. For example:<br/> <a href="https://testing.hpdd.intel.com/test_sets/cc484e04-d61a-11e7-a066-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cc484e04-d61a-11e7-a066-52540065bddc</a></p> <blockquote> <p>For ldiskfs, the error only occurred 3 times since Oct 1, 2016, all are related to <a href="https://review.whamcloud.com/#/c/28847/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28847/</a> for <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a>.</p></blockquote> <p>The failure happened on ZFS even if without such patch.</p> <p>This has failed 16x in the past week.</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/eb616ebc-e79d-11e7-9c63-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/eb616ebc-e79d-11e7-9c63-52540065bddc</a></p> <p>This is affecting patch testing on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/cbe4e4b4-e76c-11e7-9c63-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cbe4e4b4-e76c-11e7-9c63-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/3ae4250e-e917-11e7-8027-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3ae4250e-e917-11e7-8027-52540065bddc</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/444ea2f8-e62a-11e7-a066-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/444ea2f8-e62a-11e7-a066-52540065bddc</a></p> <p>+1, master<br/> <a href="https://testing.hpdd.intel.com/test_sets/68ef3b5a-e99e-11e7-a066-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/68ef3b5a-e99e-11e7-a066-52540065bddc</a></p> <p>I have looked at the logs of these failed tests, and the space usage difference is the related to the recordsize of underlying ZFS,<br/> the maximum value observed up to now is 2*recordsize, the recordsize is set to 1M (the default is 128K),</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Dec 21 2017 02:24:43.426357505 sysevent.fs.zfs.history_event version = 0x0 class = "sysevent.fs.zfs.history_event" pool = "lustre-ost2" pool_guid = 0x1c4c49fe69e964bf pool_state = 0x0 pool_context = 0x0 history_hostname = "onyx-30vm11.onyx.hpdd.intel.com" history_dsname = "lustre-ost2/ost2" history_internal_str = "recordsize=1048576" history_internal_name = "set" history_dsid = 0x88 history_txg = 0xc history_time = 0x5a3b1b6b time = 0x5a3b1b6b 0x1969b301 eid = 0x20 </pre> </div></div> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30678" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30678</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> test: limit recordsize<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d0e579ad09ac1b61509fa7a57470a48d1aed728d</p> <p>+1 on master at <a href="https://testing.hpdd.intel.com/test_sets/577b494a-f40c-11e7-8c43-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/577b494a-f40c-11e7-8c43-52540065bddc</a></p> <p>+1 on master at <a href="https://testing.hpdd.intel.com/test_sets/41d6a8c8-f614-11e7-94c7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/41d6a8c8-f614-11e7-94c7-52540065bddc</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30678/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30678/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> tests: wait for OST objects to be deleted<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1eae3bfd5de6eecbe70d24681890ad070e8446f8</p> <p>Landed for 2.11</p> <p>Hongchao - I think this issue is still open. There are a few replay-single test 20b failures that have the patch for this ticket, <a href="https://review.whamcloud.com/30678" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30678</a>, applied.</p> <p>Please see the following logs for one such failure:<a href="https://testing.hpdd.intel.com/test_sets/bbe57164-f9e6-11e7-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bbe57164-f9e6-11e7-bd00-52540065bddc</a></p> <p>Thank you</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30916" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30916</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> test: relate fs_log_size to recordsize<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c845b4230bbc4d67339c20f64187c6f4ee166275</p> <p>+1 on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/5f5fa6f4-fe4a-11e7-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5f5fa6f4-fe4a-11e7-bd00-52540065bddc</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/649ca34c-016a-11e8-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/649ca34c-016a-11e8-bd00-52540065bddc</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30916/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30916/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> test: relate fs_log_size to recordsize<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 804ea3abc4265c035d2b3941400c842e6d6fdb96</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31145" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31145</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> tests: wait for OST objects to be deleted<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: b8a94fd40ca7b829917ad437995de9091a447001</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31146" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31146</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> test: relate fs_log_size to recordsize<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: c55730b2c3aecedacababebee1437132f2820645</p> <p>Plus one on master:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/e93e1854-4731-4a95-b047-fe08c34122a5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/e93e1854-4731-4a95-b047-fe08c34122a5</a></p> <p>Anyone familiar with this able to check if it's the same issue? It is the same test failure reported here.</p> <p>Patrick,<br/> <a href="https://testing.hpdd.intel.com/test_sessions/e93e1854-4731-4a95-b047-fe08c34122a5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/e93e1854-4731-4a95-b047-fe08c34122a5</a><br/> fails with <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> because: <br/> <a href="https://review.whamcloud.com/#/c/30405/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30405/</a> parent: 2b13cb3c which does not contain <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> fix 804ea3</p> <p>Note that the commit message for the 31146 patch says that it fixes <tt>replay-single test_89</tt>, but it doesn't. The patch <a href="https://review.whamcloud.com/31120" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31120</a> "<a href="https://jira.whamcloud.com/browse/LU-5761" title="replay-single test_89: @@@@@@ FAIL: 2560 blocks leaked" class="issue-link" data-issue-key="LU-5761"><del>LU-5761</del></a> tests: fix test_89 to use fs_log_size()" is still needed to fix that test to use <tt>fs_log_size()</tt> so that it allows enough leeway in the space allocation for ZFS log blocks.</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31145/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31145/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> tests: wait for OST objects to be deleted<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: e1c3628ae4363cdb3cd28d8dc89459df819d134c</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31146/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31146/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10052" title="replay-single test_20b fails with 'after 4096 > before 3072'" class="issue-link" data-issue-key="LU-10052"><del>LU-10052</del></a> test: relate fs_log_size to recordsize<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 86cea21001f0152f6849fe5710112bced629f5f3</p> Duplicate LU-2012 Related LU-10793 LU-5761 LU-8672 LU-9891 Development Rank 1|hzzl27: Rank (Obsolete) 9223372036854775807 Severity