https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2366 Lustre <p>In the fsync at after writing in a file-per-process ior, we're seeing many lustre clients on Sequoia getting stuck in soft lockups.</p> <p>See attached console log for one of the nodes in file RB5-ID-J02.log.</p> <p>This may be a similar problem to <a href="https://jira.whamcloud.com/browse/LU-2332" title="&quot;Unable to handle kernel paging request&quot; in osc_queue_sync_pages" class="issue-link" data-issue-key="LU-2332"><del>LU-2332</del></a>, but under a user-space process, not under ptlrpcd. </p> 2.3.56-2chaos (github.com/chaos/lustre) LU-2366

Soft lockups under ptlrpc_check_set

Bug Major Resolved Duplicate Jinshan Xiong Christopher Morrone sequoia Tue, 20 Nov 2012 17:32:08 +0000 Mon, 3 Dec 2012 17:13:00 +0000 Mon, 3 Dec 2012 17:13:00 +0000 Lustre 2.4.0 0 7 <p>Alex will triage</p> <p>please have a look</p> <p>can you please show me the source code at ptlrpc_check_set+0x4f4/0x4e80?</p> <p>Thanks.</p> <p>This looks similar to what we saw in yesterday's testing, which I commented on in <a href="https://jira.whamcloud.com/browse/LU-2263" title="CPU Soft Lockups due to many threads spinning on import lock on Sequoia IO nodes" class="issue-link" data-issue-key="LU-2263"><del>LU-2263</del></a> <a href="http://jira.whamcloud.com/browse/LU-2263?focusedCommentId=48540&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-48540" class="external-link" rel="nofollow">here</a>.</p> <blockquote> <p>can you please show me the source code at ptlrpc_check_set+0x4f4/0x4e80?</p></blockquote> <p>This was taken from modules and code built from our <tt>2.3.56-3chaos</tt> tag (the log is running <tt>2.3.56-2chaos</tt>, but I don't think <tt>ptlrpc_check_set</tt> has changed so:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(gdb) l *ptlrpc_check_set+0x4f4 0x46ed4 is in ptlrpc_check_set (/builddir/build/BUILD/lustre-2.3.56/lustre/ptlrpc/client.c:1852). 1847 /builddir/build/BUILD/lustre-2.3.56/lustre/ptlrpc/client.c: No such file or directory. in /builddir/build/BUILD/lustre-2.3.56/lustre/ptlrpc/client.c </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1849 libcfs_nid2str(imp-&gt;imp_connection-&gt;c_peer.nid), 1850 lustre_msg_get_opc(req-&gt;rq_reqmsg)); 1851 1852 cfs_spin_lock(&amp;imp-&gt;imp_lock); 1853 /* Request already may be not on sending or delaying list. This 1854 * may happen in the case of marking it erroneous for the case 1855 * ptlrpc_import_delay_req(req, status) find it impossible to </pre> </div></div> <p>I think this is the same problem to <a href="https://jira.whamcloud.com/browse/LU-2263" title="CPU Soft Lockups due to many threads spinning on import lock on Sequoia IO nodes" class="issue-link" data-issue-key="LU-2263"><del>LU-2263</del></a> and the root cause is high contention on import lock. Just in case, I'll leave this ticket open but lower the priority.</p> <p>As per LLNL, close and reopen if this reoccurs</p> Related LU-2263 Development Rank 1|hzvclj: Rank (Obsolete) 5629 Severity