https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5039 Lustre <p>Running Lustre 2.4.0-28chaos (see github.com/chaos/lustre), we find that sometimes after a reboot the MDS can get stuck during mount cleaning up the orphan files in the PENDING directory. Some times we have 100,000+ files to process, and this can take literally hours. The symptoms are pretty similar to <a href="https://jira.whamcloud.com/browse/LU-5038" title="Mount hangs for hours processing some llog" class="issue-link" data-issue-key="LU-5038"><del>LU-5038</del></a>, but I believe that the cause is different.</p> <p>Here is a backtrace of the offending thread:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2014-03-06 22:34:12 Process tgt_recov (pid: 15478, threadinfo ffff8807bc436000, task ffff88081a6e2080) 2014-03-06 22:34:12 Stack: 2014-03-06 22:34:12 ffff88072e3df000 0000000000000000 0000000000003f14 ffff88072e3df060 2014-03-06 22:34:12 &lt;d&gt; ffff8807bc437a40 ffffffffa0341396 ffff8807bc437a20 ffff88072e3df038 2014-03-06 22:34:12 &lt;d&gt; 0000000000000014 ffff8807f9fbf530 0000000000000000 0000000000003f14 2014-03-06 22:34:12 Call Trace: 2014-03-06 22:34:12 [&lt;ffffffffa0341396&gt;] __dbuf_hold_impl+0x66/0x480 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa034182f&gt;] dbuf_hold_impl+0x7f/0xb0 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa03428e0&gt;] dbuf_hold+0x20/0x30 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa03486e7&gt;] dmu_buf_hold+0x97/0x1d0 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa03369a0&gt;] ? remove_reference+0xa0/0xc0 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa039e76b&gt;] zap_idx_to_blk+0xab/0x140 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa039ff61&gt;] zap_deref_leaf+0x51/0x80 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa039f956&gt;] ? zap_put_leaf+0x86/0xe0 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa03a03dc&gt;] fzap_cursor_retrieve+0xfc/0x2a0 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa03a593b&gt;] zap_cursor_retrieve+0x17b/0x2f0 [zfs] 2014-03-06 22:34:12 [&lt;ffffffffa0d1739c&gt;] ? udmu_zap_cursor_init_serialized+0x2c/0x30 [osd_zfs] 2014-03-06 22:34:12 [&lt;ffffffffa0d29058&gt;] osd_index_retrieve_skip_dots+0x28/0x60 [osd_zfs] 2014-03-06 22:34:12 [&lt;ffffffffa0d29638&gt;] osd_dir_it_next+0x98/0x120 [osd_zfs] 2014-03-06 22:34:12 [&lt;ffffffffa0f08161&gt;] lod_it_next+0x21/0x90 [lod] 2014-03-06 22:34:12 [&lt;ffffffffa0dd1989&gt;] __mdd_orphan_cleanup+0xa9/0xca0 [mdd] 2014-03-06 22:34:12 [&lt;ffffffffa0de134d&gt;] mdd_recovery_complete+0xed/0x170 [mdd] 2014-03-06 22:34:12 [&lt;ffffffffa0e34cb5&gt;] mdt_postrecov+0x35/0xd0 [mdt] 2014-03-06 22:34:12 [&lt;ffffffffa0e36178&gt;] mdt_obd_postrecov+0x78/0x90 [mdt] 2014-03-06 22:34:12 [&lt;ffffffffa08745c0&gt;] ? ldlm_reprocess_res+0x0/0x20 [ptlrpc] 2014-03-06 22:34:12 [&lt;ffffffffa086f8ae&gt;] ? ldlm_reprocess_all_ns+0x3e/0x110 [ptlrpc] 2014-03-06 22:34:12 [&lt;ffffffffa0885004&gt;] target_recovery_thread+0xc64/0x1980 [ptlrpc] 2014-03-06 22:34:12 [&lt;ffffffffa08843a0&gt;] ? target_recovery_thread+0x0/0x1980 [ptlrpc] 2014-03-06 22:34:12 [&lt;ffffffff8100c10a&gt;] child_rip+0xa/0x20 2014-03-06 22:34:12 [&lt;ffffffffa08843a0&gt;] ? target_recovery_thread+0x0/0x1980 [ptlrpc] 2014-03-06 22:34:12 [&lt;ffffffffa08843a0&gt;] ? target_recovery_thread+0x0/0x1980 [ptlrpc] 2014-03-06 22:34:12 [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>The mount process is blocked while this is going on. The cleanup is completely sequential and on ZFS very slow, on the order of 10 per second.</p> <p>The orphan cleanup task really needs to be backgrounded (and perhaps parallelized) rather than blocking the MDT mount processes.</p> LU-5039

MDS mount hangs on orphan recovery

Bug Major Resolved Fixed Niu Yawei Christopher Morrone mn4 Fri, 9 May 2014 21:11:08 +0000 Tue, 12 Aug 2014 19:43:52 +0000 Mon, 23 Jun 2014 21:42:35 +0000 Lustre 2.6.0 Lustre 2.5.3 0 6 <p>Alex</p> <p>I think that it is best that you comment on this one</p> <p>Peter</p> <p>the idea is correct and fine. though I'm very confused by 10/second - we should be able to do much faster, given no LDLM contention, etc.</p> <p>Niu </p> <p>Could you please create a patch based on Oleg's suggestion (to follow)</p> <p>Thanks</p> <p>Peter</p> <p>While these slow deletions are extreme and at least we can speed up the startup by doing deetions from a separate thread once reovery is complete.<br/> Basiclly we'll create a new PENDING dir and will move all entries fclaimed by recovery rom old pending there. Then we just spawn another thread to delete the old pending and its content.</p> <p>Need to be careful about of mds failure while doing this split handling and another recovery-restart - we probably would need to move all entries to old pending and redo the process. there sohul not be many due to recovery I hope</p> <blockquote> <p>Basiclly we'll create a new PENDING dir and will move all entries fclaimed by recovery rom old pending there. Then we just spawn another thread to delete the old pending and its content.<br/> Need to be careful about of mds failure while doing this split handling and another recovery-restart - we probably would need to move all entries to old pending and redo the process. there sohul not be many due to recovery I hope</p></blockquote> <p>What bad will happen if we just start a thread to delete orphans from the original PENDING?</p> <p>the possible problem is that list of items in PENDING is not fixed, new files might be added.<br/> Can we reliably and race-free tell the ones that are still needed from those that are stale and need to be killed? <br/> Also things like NFS further omplicate thingsby possible brifly reattaching to deleted files that were ought to be deleted.</p> <p>there is open count in mdd object which tells whether the file is in use. probably we'll have to add locking to protect the last close vs. the cleanup procedure..</p> <blockquote> <p>there is open count in mdd object which tells whether the file is in use. probably we'll have to add locking to protect the last close vs. the cleanup procedure..</p></blockquote> <p>Indeed, I've talked with Oleg about this, and looks we already have lock serialized last close and orphan cleanup. I'll compose patch soon. Thank you all.</p> <p>cleanup orphan asynchronously: <a href="http://review.whamcloud.com/10584" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10584</a></p> <p>patch landed on master, need we backport it b2_4 &amp; b2_5?</p> <p>Yes I thin that we should</p> <p>b2_4: <a href="http://review.whamcloud.com/10779" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10779</a><br/> b2_5: <a href="http://review.whamcloud.com/10780" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10780</a></p> <p>Landed for 2.6</p> Development Rank 1|hzwm87: Rank (Obsolete) 13931 Severity