https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8509 Lustre <p>Running lustre 2.8.0_0.0.llnlpreview.18 on the clients (see the lustre-release-fe-llnl) , we are regularly seeing hangs of the /etc/slurm/prolog script when it triggers drop_caches. This script runs before each job to clear out the cache from any previous jobs.</p> <p>In particular it hangs here:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># Flush slab cache entries echo 2 &gt;/proc/sys/vm/drop_caches </pre> </div></div> <p>And this is the backtrace for where it is getting stuck:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; bt -xs 1386 PID: 1386 TASK: ffff88201b0a5080 CPU: 10 COMMAND: "prolog" #0 [ffff882011bd3af8] __schedule+0x295 at ffffffff81651975 #1 [ffff882011bd3b60] schedule+0x29 at ffffffff81652049 #2 [ffff882011bd3b70] cl_inode_fini+0x1ac at ffffffffa0c6b3ac [lustre] #3 [ffff882011bd3c10] ll_clear_inode+0x21c at ffffffffa0c377ec [lustre] #4 [ffff882011bd3c38] ll_delete_inode+0x58 at ffffffffa0c39048 [lustre] #5 [ffff882011bd3c60] evict+0xa7 at ffffffff81204077 #6 [ffff882011bd3c88] dispose_list+0x3e at ffffffff8120417e #7 [ffff882011bd3cb0] prune_icache_sb+0x163 at ffffffff81205113 #8 [ffff882011bd3d18] prune_super+0x143 at ffffffff811ea343 #9 [ffff882011bd3d50] shrink_slab+0x175 at ffffffff81183a25 #10 [ffff882011bd3e08] drop_caches_sysctl_handler+0x283 at ffffffff8124a743 #11 [ffff882011bd3e90] proc_sys_call_handler+0xd3 at ffffffff81260f03 #12 [ffff882011bd3ee8] proc_sys_write+0x14 at ffffffff81260f34 #13 [ffff882011bd3ef8] vfs_write+0xbd at ffffffff811e7bfd #14 [ffff882011bd3f38] sys_write+0x7f at ffffffff811e869f #15 [ffff882011bd3f80] system_call_fastpath+0x16 at ffffffff8165d709 RIP: 00007ffff76d3500 RSP: 00007fffffffe180 RFLAGS: 00010206 RAX: 0000000000000001 RBX: ffffffff8165d709 RCX: 0000000000000400 RDX: 0000000000000002 RSI: 00007ffff7ff8000 RDI: 0000000000000001 RBP: 00007ffff7ff8000 R8: 000000000000000a R9: 00007ffff7fbd740 R10: 00007fffffffe670 R11: 0000000000000246 R12: 0000000000000001 R13: 0000000000000002 R14: 00007ffff79a7400 R15: 0000000000000002 ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b </pre> </div></div> LU-8509

drop_caches hangs in cl_inode_fini()

Bug Major Resolved Fixed Zhenyu Xu Christopher Morrone llnl Wed, 17 Aug 2016 00:08:58 +0000 Mon, 6 Jan 2020 22:19:14 +0000 Wed, 5 Oct 2016 11:57:07 +0000 Lustre 2.8.0 Lustre 2.9.0 0 13 <p>Bobijam</p> <p>Could you please asssit on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Do the image built with --enable-lu_ref defined in configure? cl_inode_fini() is waiting for the lli_clob reference becoming to 1, and it seems that another thread referenced the object does not release the reference thereafter.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Do the image built with --enable-lu_ref defined in configure?</pre> </div></div> <p>No, we are not setting that.</p> <p>Can you dump all threads trace of this hit? (echo t &gt; /proc/sysrq_trigger)</p> <p>Perhaps the next time it happens.</p> <p>Cray has hit this bug several times. apinit, one of the Cray workload manager daemons, hangs while dropping vm caches: echo 3 &gt; /proc/sys/vm/drop_caches. apinit drops vm caches at the end of each job after dropping ldlm caches. When apinit fails to complete the vm drop_caches, the Node Health Checker (NHC) first marks the node suspect and then marks it admindown. In this state no new jobs are scheduled.</p> <p>apinit appears to be hung waiting for the loh_ref count of a cl_object to drop from 2 to 1.</p> <p>I uploaded a dump to ftp.intel.com:/uploads/<a href="https://jira.whamcloud.com/browse/LU-8509" title="drop_caches hangs in cl_inode_fini()" class="issue-link" data-issue-key="LU-8509"><del>LU-8509</del></a> in case it may be of some help.</p> <p>Andrew Perepechko (andrew.perepechko@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/22743" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22743</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8509" title="drop_caches hangs in cl_inode_fini()" class="issue-link" data-issue-key="LU-8509"><del>LU-8509</del></a> tests: drop_caches hangs in cl_inode_fini()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9b1a256b2cede7866fa7c86916ddebab88800ad0</p> <p>Andrew Perepechko (andrew.perepechko@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/22745" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22745</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8509" title="drop_caches hangs in cl_inode_fini()" class="issue-link" data-issue-key="LU-8509"><del>LU-8509</del></a> llite: drop_caches hangs in cl_inode_fini()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 124b249c8ccaa4aba925916752d0a3fa51fda2f1</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22745/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22745/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8509" title="drop_caches hangs in cl_inode_fini()" class="issue-link" data-issue-key="LU-8509"><del>LU-8509</del></a> llite: drop_caches hangs in cl_inode_fini()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c594026329e6a78a6c9f3188514211647b3040d8</p> <p>Landed for 2.9</p> Duplicate LU-8936 LU-8743 Related LU-8743 Development Rank 1|hzykyn: Rank (Obsolete) 9223372036854775807 Severity