https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10913 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/d0b6dfde-3eaf-11e8-960d-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d0b6dfde-3eaf-11e8-960d-52540065bddc</a></p> <p>test_106 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 276 mins, last suite running was conf-sanity, restarting cluster to continue tests </pre> </div></div> <p>MDS error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 12 21:13:22 trevis-72vm6 kernel: Lustre: 24256:0:(llog_cat.c:926:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Apr 12 21:13:29 trevis-72vm6 kernel: LustreError: 24256:0:(llog_cat.c:269:llog_cat_id2handle()) lustre-OST0000-osc-MDT0000: error opening log id [0xa:0x1:0x0]:0: rc = -2 Apr 12 21:13:29 trevis-72vm6 kernel: LustreError: 24256:0:(llog_cat.c:795:llog_cat_cancel_records()) lustre-OST0000-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 Apr 12 21:13:29 trevis-72vm6 kernel: LustreError: 24256:0:(osp_sync.c:1062:osp_sync_process_committed()) lustre-OST0000-osc-MDT0000: can't cancel record: -2 </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> conf-sanity test_106 - Timeout occurred after 276 mins, last suite running was conf-sanity, restarting cluster to continue tests</p> LU-10913

conf-sanity test_106: OST0000-osc-MDT0000: can't cancel record

Bug Minor Resolved Fixed Alexander Boyko Maloo patch Fri, 13 Apr 2018 18:59:07 +0000 Mon, 8 Apr 2019 13:28:00 +0000 Mon, 11 Feb 2019 15:12:38 +0000 Lustre 2.13.0 Lustre 2.12.1 0 4 <p>I've seen something similar for 106</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 00000004:00080000:2.0:1542884025.062076:0:11650:0:(osp_sync.c:510:osp_sync_request_commit_cb()) commit req ffff8800123e3300, transno 0 00000004:00080000:3.0:1542884025.062076:0:11649:0:(osp_sync.c:510:osp_sync_request_commit_cb()) commit req ffff8800123e3600, transno 0 00000004:00080000:2.0:1542884025.062077:0:11650:0:(osp_sync.c:564:osp_sync_interpret()) reply req ffff8800123e3300/1, rc -2, transno 0 00000004:00080000:3.0:1542884025.062077:0:11649:0:(osp_sync.c:564:osp_sync_interpret()) reply req ffff8800123e3600/1, rc -2, transno 0 00000004:00001000:2.0:1542884025.062079:0:11650:0:(osp_sync.c:620:osp_sync_interpret()) lustre-OST0000-osc-MDT0000: 7 in flight, 3985 in progress 00000004:00001000:3.0:1542884025.062079:0:11649:0:(osp_sync.c:620:osp_sync_interpret()) lustre-OST0000-osc-MDT0000: 6 in flight, 3985 in progress 00000100:00001000:2.0:1542884025.062081:0:11650:0:(import.c:1695:at_measured()) add 1 to ffff88002a111c00 time=131 v=1 (1 0 0 0) 00000100:00001000:3.0:1542884025.062081:0:11649:0:(import.c:1695:at_measured()) add 1 to ffff88002a111c00 time=131 v=1 (1 0 0 0) 00000100:00001000:2.0:1542884025.062083:0:11650:0:(import.c:1695:at_measured()) add 1 to ffff88002a111bc8 time=131 v=1 (1 0 0 0) 00000100:00001000:3.0:1542884025.062083:0:11649:0:(import.c:1695:at_measured()) add 1 to ffff88002a111bc8 time=131 v=1 (1 0 0 0) 00000004:00080000:2.0:1542884025.062084:0:11650:0:(osp_sync.c:510:osp_sync_request_commit_cb()) commit req ffff8800123e3900, transno 0 00000004:00080000:2.0:1542884025.062085:0:11650:0:(osp_sync.c:564:osp_sync_interpret()) reply req ffff8800123e3900/1, rc -2, transno 0 00000004:00080000:3.0:1542884025.062085:0:11649:0:(osp_sync.c:510:osp_sync_request_commit_cb()) commit req ffff8800123e3000, transno 0 00000004:00001000:2.0:1542884025.062086:0:11650:0:(osp_sync.c:620:osp_sync_interpret()) lustre-OST0000-osc-MDT0000: 5 in flight, 3985 in progress 00000004:00080000:3.0:1542884025.062086:0:11649:0:(osp_sync.c:564:osp_sync_interpret()) reply req ffff8800123e3000/1, rc -2, transno 0</pre> </div></div> <p>It means the request with unlink/etc. was sent to OST and OST reported no object exist(ENOENT).<br/> After capturing couple vmcores with different logging options I've found that last catalog record was processed twice.<br/> First time for a normal processing and second time when wrapping.<br/> osp_sync_thread() using opd_sync_catalog_idx to indicate the start point for processing. <br/> rc = llog_cat_process(&amp;env, llh, osp_sync_process_queues, d, <br/> d-&gt;opd_sync_last_catalog_idx, 0);<br/> But it is ignored when catalog is wrapped at the llog_cat_process_or_fork(). It starts from the the end of wrapping. For example catalog records is <br/> 1 1 0 0 1.<br/> The processing would start from record 5.<br/> But normal processing was stopped at the end of catalog, so we should skip the last record and start from a first one.</p> <p>Alexandr Boyko (c17825@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/33749" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33749</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10913" title="conf-sanity test_106: OST0000-osc-MDT0000: can&#39;t cancel record" class="issue-link" data-issue-key="LU-10913"><del>LU-10913</del></a> llog: add startcat for wrapped catalog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e6b3eae5b5532e48405fc485f7a025a91089dd2d</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33749/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33749/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10913" title="conf-sanity test_106: OST0000-osc-MDT0000: can&#39;t cancel record" class="issue-link" data-issue-key="LU-10913"><del>LU-10913</del></a> llog: add startcat for wrapped catalog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8109c9e1718d7f425a7186c31399ac035157acd6</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34506" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34506</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10913" title="conf-sanity test_106: OST0000-osc-MDT0000: can&#39;t cancel record" class="issue-link" data-issue-key="LU-10913"><del>LU-10913</del></a> llog: add startcat for wrapped catalog<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: aed0d8a71803ec19d80e4adaa33bab2b09273a9c</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34506/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34506/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10913" title="conf-sanity test_106: OST0000-osc-MDT0000: can&#39;t cancel record" class="issue-link" data-issue-key="LU-10913"><del>LU-10913</del></a> llog: add startcat for wrapped catalog<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: fe0f046d85d1c8f415e7744f4ff7de611efdbff0</p> Development Rank 1|hzzvuf: Rank (Obsolete) 9223372036854775807 Severity