https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11857 Lustre <p>After upgrade from 2.9 to 2.12 the MDS syslog gets beaucoup error messages <em><b>lfs-MDD0000 cannot delete orphan messages</b></em>. On the active MDS the <em><b>orph_lfs-MDD00</b></em> process is pegged at 100%</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[Sat Jan 12 23:21:18 2019] LustreError: 14125:0:(mdd_orphans.c:327:mdd_orphan_destroy()) lfs-MDD0000: could not delete orphan [0x200060151:0x38a8:0x0]: rc = -2 [Sat Jan 12 23:21:18 2019] LustreError: 14125:0:(mdd_orphans.c:327:mdd_orphan_destroy()) Skipped 8067628 previous similar messages [Sat Jan 12 23:31:18 2019] LustreError: 14125:0:(mdd_orphans.c:327:mdd_orphan_destroy()) lfs-MDD0000: could not delete orphan [0x200060151:0x38a8:0x0]: rc = -2 [Sat Jan 12 23:31:18 2019] LustreError: 14125:0:(mdd_orphans.c:327:mdd_orphan_destroy()) Skipped 7958773 previous similar messages [Sat Jan 12 23:41:18 2019] LustreError: 14125:0:(mdd_orphans.c:327:mdd_orphan_destroy()) lfs-MDD0000: could not delete orphan [0x200060151:0x38a8:0x0]: rc = -2 </pre> </div></div> LU-11857

repeated "could not delete orphan [0x200060151:0x38a8:0x0]: rc = -2" messages

Bug Minor Resolved Duplicate Alex Zhuravlev Jeff Johnson Mon, 14 Jan 2019 19:01:24 +0000 Sat, 16 Feb 2019 16:36:09 +0000 Sat, 16 Feb 2019 16:36:03 +0000 0 5 <p>Alex</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>Update:  Performed a full shutdown of the file system and all server systems.  Rebooted and performed orderly start of the file system.</p> <p>The reported <em><b>orph_lfs-MDD00 thread</b></em> at 100% CPU and</p> <p>syslog entries of </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 109872:0:(mdd_orphans.c:327:mdd_orphan_destroy()) ls15-MDD0000: could not delete orphan [0x200060151:0x38a8:0x0] </pre> </div></div> <p>continue to occur. After 15 minutes of wall time with the file system targets mounted the errors persist.</p> <p>Let me know if you want me to upload another dump of the debug buffer.</p> <p>In 13 minutes the <b><em>mdd_orphans.c:327:mdd_orphan_destroy</em></b> have incremented quantity from 1979094 to 8056841  (6077747 syslog msg repeats in 13 minutes).</p> <p> </p> <p>After seven hours the <b><em>orph_lfs-MDD00</em></b> thread still pegged at 100%</p> <p>Attaching <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31780/31780_orph_100pct_201901142137.txt.bz2" title="orph_100pct_201901142137.txt.bz2 attached to LU-11857">orph_100pct_201901142137.txt.bz2^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>I think <a href="https://review.whamcloud.com/#/c/33661/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33661/</a> should help</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=aeonjeffj" class="user-hover" rel="aeonjeffj">aeonjeffj</a>can you please try <a href="https://review.whamcloud.com/#/c/33661/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33661/</a> ?</p> <p>I can. This is a production LFS. Given that there is data in place, should I? Not arguing, just applying caution and respect for end user's data.</p> <p>The patch has passed review and testing and is scheduled to land in 2.13 shortly. This should avoid the repeated attempts to destroy the same object.</p> <p>So, it seems like this is believed to be a duplicate of the recently landed <a href="https://jira.whamcloud.com/browse/LU-11418" title="hung threads on MDT and MDT won&#39;t umount" class="issue-link" data-issue-key="LU-11418"><del>LU-11418</del></a> fix. Jeff, if this issue is not currently causing MSU any heartburn and explicitly trying to prove/disprove whether the theory is sound would be disruptive, is it enough to close this ticket as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-11418" title="hung threads on MDT and MDT won&#39;t umount" class="issue-link" data-issue-key="LU-11418"><del>LU-11418</del></a> and reopen it if it is seen on a release including the fix (2.13, or an upcoming 2.12.x maintenance releases)</p> <p>No objections it seems</p> Related LU-11418 Development Rank 1|i009hz: Rank (Obsolete) 9223372036854775807 Severity