https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8367 Lustre <p>problem discovered while testing a OST failovers. OST pool with 10 OST was created and striping with -1 assigned to it.<br/> half (even indexes) OST's have failed during create.<br/> object creation was blocked in several places, sometimes after reserving an object on failed OST. In that case OSP threads was blocked to start a delete orphans due situation when allocation hold an some reserved objects and can't be release this reservation due blocking on waiting recovery on next assigned OST. Due some object allocations in parallel - MDT hit in situation when each failed OST have an own reserved object and objects allocation blocked by long time waiting a specially when all OSP timeouts (each obd_timeout) expired. It may need a large amount of time - half or full hour.</p> <p>That bug introduced as regression after LOV &gt; LOD moving on MDT side.<br/> Original ticket is <a href="https://projectlava.xyratex.com/show_bug.cgi?id=18357" class="external-link" target="_blank" rel="nofollow noopener">https://projectlava.xyratex.com/show_bug.cgi?id=18357</a></p> LU-8367

delete orphan phase isn't stated for multistriped file

Bug Critical Resolved Fixed Alex Zhuravlev Alexey Lyashkov Tue, 5 Jul 2016 09:36:44 +0000 Fri, 24 Nov 2023 06:16:35 +0000 Tue, 3 Jan 2023 21:44:02 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.16.0 0 13 <p>problem easy replicated with attached test.</p> <p>main problem is not about del orphan start, but recovery block stop any allocation until del orphan finished, so one failed create will block any allocations.</p> <p>Alexander Boyko (alexander.boyko@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/21483" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21483</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a> test: long waiting for multistripe file creation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d23f8ae00fc85a7fe070a72f35cc90a1f036afb3</p> <p>Suggested solution <a href="http://review.whamcloud.com/#/c/21785/4" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/21785/4</a> doesn't solve the issue when reconnection is caused by OST failover.<br/> I changed reproducer <a href="http://review.whamcloud.com/21483" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21483</a> (patchset 4). Now it shows hunging osp_precreate_cleanup_orphans after OST failover.</p> <p>Hi Peter, Vitaly,</p> <p>I think this one is also stuck due to mismatch in expectations. Is it possible to quickly summarize from each team what the expectations are so we can try to move this forward?</p> <p>Regards,<br/> Tejas.</p> <p>We discussed the issue at LAD, I've been working on the solution..</p> <p>Great!! .. Thanks ..</p> <p><a href="http://review.whamcloud.com/#/c/23168/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/23168/</a></p> <p>May you explain why it hack better to disable a del orphan expect an MDT recovery phase ?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/23168/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23168/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a> osp: do not block orphan cleanup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2ce0d5b0640e3e440822080e407eee1ce1cafd75</p> <p>Landed for 2.10</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/25925" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25925</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a> osp: do not block orphan cleanup"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 27bdaf74982212f9441cefd28cb533289c0f7bfe</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25926/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25926/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a> osp: orphan cleanup do not wait for reserved<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1b3028ab142a1f605e6274a6019bb39d89ae8d25</p> <p>Reverted from master under <a href="https://jira.whamcloud.com/browse/LU-9498" title="osp_precreate_get_fid()) ASSERTION( osp_fid_diff(&amp;d-&gt;opd_pre_used_fid, &amp;d-&gt;opd_pre_last_created_fid) &lt; 0 ) failed: next fid [0x680000402:0x25de031:0x0] last created fid [0x680000402:0x25de031:0x0]" class="issue-link" data-issue-key="LU-9498"><del>LU-9498</del></a></p> <p>This issue is actual still with Lustre 2.15.  We do have a vmcore with MDT failover problem. However r/c brings me to this issue. The test did not include massive failover as description has, only one by one  random node failover/failback during IO.</p> <blockquote><p><span class="error">&#91;18971.201645&#93;</span> LustreError: 0-0: Forced cleanup waiting for mdt-kjcf05-MDT0001_UUID namespace with 46 resources in use, (rc=-110)</p></blockquote> <p>MDT01 creation threads hanged on lq_rw_sem 27 threads sleeped about 1.5 hours<br/> <span class="error">&#91;0 01:29:59.961&#93;</span> <span class="error">&#91;UN&#93;</span>  PID: 49816  TASK: ffff98c656af17c0  CPU: 5   COMMAND: "mdt01_000"<br/> , lod_ost_alloc_rr holded semaphore for read by 3 threads <br/> <span class="error">&#91;0 00:05:00.729&#93;</span> <span class="error">&#91;ID&#93;</span>  PID: 56895  TASK: ffff98c67f1217c0  CPU: 22  COMMAND: "mdt05_007"<br/> They waked up every obdtimeout (5minutes), and did not go further because of no objects.<br/> The next msg repeated during 4000+ seconds.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00080000:7.0:1644534589.764767:0:90378:0:(osp_precreate.c:1529:osp_precreate_reserve()) kjcf05-OST0003-osc-MDT0001: slow creates, last=[0x340000400:0x23a4f483:0x0], next=[0x340000400:0x23a4f378:0x0], reserved=267, sync_changes=1, sync_rpcs_in_progress=0, status=-19  .... 00000004:00080000:3.0:1644539373.892843:0:84791:0:(osp_precreate.c:1529:osp_precreate_reserve()) kjcf05-OST0003-osc-MDT0001: slow creates, last=[0x340000400:0x23a4f483:0x0], next=[0x340000400:0x23a4f378:0x0], reserved=267, sync_changes=0, sync_rpcs_in_progress=0, status=0 </pre> </div></div> <p>During this time period MDT0001 successfully reconnected to OST0003, it have done recovery etc.  four times. But objects were not created.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:02000000:26.0:1644534989.327955:0:49612:0:(import.c:1642:ptlrpc_import_recovery_state_machine()) kjcf05-OST0003-osc-MDT0001: Connection restored to 10.16.100.58@o2ib (at 10.16.100.58@o2ib) 00000100:02000000:26.0:1644536000.579884:0:49612:0:(import.c:1642:ptlrpc_import_recovery_state_machine()) kjcf05-OST0003-osc-MDT0001: Connection restored to 10.16.100.60@o2ib (at 10.16.100.60@o2ib) 00000100:02000000:27.0:1644537071.966959:0:49612:0:(import.c:1642:ptlrpc_import_recovery_state_machine()) kjcf05-OST0003-osc-MDT0001: Connection restored to 10.16.100.58@o2ib (at 10.16.100.58@o2ib) 00000100:02000000:24.0:1644538459.546055:0:49612:0:(import.c:1642:ptlrpc_import_recovery_state_machine()) kjcf05-OST0003-osc-MDT0001: Connection restored to 10.16.100.60@o2ib (at 10.16.100.60@o2ib) </pre> </div></div> <p>osp_precreate_thread sleeps here</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 49938 TASK: ffff98c63a248000 CPU: 30 COMMAND: "osp-pre-3-1" #0 [ffffa545b38efd48] __schedule at ffffffff8e54e1d4 #1 [ffffa545b38efde0] schedule at ffffffff8e54e648 #2 [ffffa545b38efdf0] osp_precreate_cleanup_orphans at ffffffffc17d00e9 [osp] #3 [ffffa545b38efe70] osp_precreate_thread at ffffffffc17d18da [osp] </pre> </div></div> <p>The wait is</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>wait_event_idle(d-&gt;opd_pre_waitq, (!d-&gt;opd_pre_reserved &amp;&amp; d-&gt;opd_recovery_completed) || !d-&gt;opd_pre_task || d-&gt;opd_got_disconnected); osp_pre_reserved = 0x10b, </pre> </div></div> <p>osp_pre_reserverd was not decreased to 0, and osp_precreate_thread was blocked at osp_precreate_cleanup_orphans()<br/> The hanged component is overstriped with 2000 stripes count.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; lod_layout_component 0xffff98c68f4a8100 struct lod_layout_component { llc_extent = { e_start = 536870912, e_end = 536870912 }, llc_id = 4, llc_flags = 0, llc_stripe_size = 1048576, llc_pattern = 513, llc_layout_gen = 0, llc_stripe_offset = 65535, llc_stripe_count = 2000, llc_stripes_allocated = 0, llc_timestamp = 0, llc_pool = 0xffff98c60eef5980 "flash", llc_ostlist = { </pre> </div></div> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bzzz" class="user-hover" rel="bzzz">bzzz</a> ^</p> <p>the root cause is the object creation which were holding some objects reserved for simple reason - orphan cleanup doesn't know which object to start cleanup from. in turn, object reservation is required to avoid waiting for object within a local transaction which would be bad.<br/> probably the object precreation should be fixed to handle overstriping "faster" when some OSTs are missing.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46889/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46889/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a> osp: enable replay for precreation request<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 63e17799a369e2ff0b140fd41dc5d7d8656d2bf0</p> <p>"Li Dongyang &lt;dongyangli@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49151" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49151</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a> osp: detect reformatted OST for FID_SEQ_NORMAL<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e39c59d6845b0bff7458fc996bdf02f9f62980a7</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49151/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49151/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a> osp: wait for precreate on reformatted OST<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e06b2ed956f45feffa3adc7e2e7399ab737b37be</p> <p>Landed for 2.16</p> <p>"Sergey Cheremencev &lt;scherementsev@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50543" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50543</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a> osp: unused fail_locs from sanity-27S<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 188103c3203355df222e7b43a45e02405bd8fe4a</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50543/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50543/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a> osp: remove unused fail_locs from sanity/27S,822<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e69eea5f60eec17ac32cea8d2a60768e0738a052</p> Blocker Duplicate Related LU-9498 LU-10336 LU-16425 LU-9285 Development Rank 1|hzyglz: Rank (Obsolete) 9223372036854775807 Severity