https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7330 Lustre <p>Our system administrators have reported that Lustre clients on our big BG/Q system are locking up due to lustre. There are some softlockups, and then the kernel reports a "spinlock lockup" and dumps backtraces for many processes that are all stuck spinning on the same ldlm blp_lock.</p> <p>Many ldlm_bl_* threads are all spinning here:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>_raw_spin_lock _spin_lock ldlm_bl_get_work ldlm_bl_thread_main </pre> </div></div> <p>Meanwhile many sysiod processes (the I/O node portion of the I/O forwarding system on BG/Q) are stuck in either a read or write system call path that go through shrink_slab() and get stuck on the same spin lock. Here is an example backtrace for a read case:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>_raw_spin_lock _spin_lock _ldlm_bl_to_thread ldlm_bl_to_thread ldlm_cancel_lru ldlm_cli_pool_shrink ldlm_pool_shrink ldlm_pools_shrink shrink_slab do_try_free_pages try_free_pages __alloc_pages_nodemask generic_file_aio_read vvp_io_read_start cl_io_start cl_io_loop ll_file_io_generic ll_file_aio_read ll_file_read vfs_read sys_read </pre> </div></div> <p>Crash dumps are unfortunately not available on this system. That is about ll the information that I can currently get.</p> <p>Those clients are running lustre 2.5.4-4chaos.</p> LU-7330

spinlock lockup on ldlm blp_lock in ldlm_bl_* threads

Bug Major Resolved Fixed Niu Yawei Christopher Morrone llnl llnlfixready Fri, 23 Oct 2015 00:39:57 +0000 Fri, 27 Jan 2017 22:38:46 +0000 Wed, 11 Nov 2015 18:14:48 +0000 Lustre 2.8.0 0 7 <p>When the clients lock up, do you know how many <tt>ldlm_bl_*</tt> threads are running? There was another issue reported recently about too many ldlm_bl threads being started when a process is interrupted, but setting <tt>options ptlrpc ldlm_num_threads=16</tt> has avoided the problem. </p> <p>The "spinlock lockup" backtraces are showing in the 50+ threads, and the thread numbers are up in the 120s. I am also seeing multiple ldlm_bl_* threads duplicate names.</p> <p>On one node that is not currently locked up, there are 160 ldlm_bl_* threads with only 72 unique names between them.</p> <p>It sounds as if this may be a similar problem. The thread names are not important for their functionality, but the duplicate names do imply that many of the threads started around the same time. You could try the module option, but that won't take effect until the next remount. </p> <p>Niu</p> <p>Could you please assist with this issue?</p> <p>Thanks</p> <p>Peter</p> <p>[ Comments from the other ticket, which cannot be made public ]</p> <blockquote> <p>Users that run jobs interactively and when they kill their jobs with Ctrl-C, they find that some nodes will end up with 100-200 LDLM processes. For comparison, freshly rebooted nodes have only 15-25 ldlm processes. The user checked and confirmed that after the Ctrl-C, there are no more processes of the user job remaining, nor are fuser and lsof reporting any open files. It is therefore not clear what is driving up the LDLM process count.</p></blockquote> <p>There is one problem that is shown by the stack traces, which may only be cosmetic, but may also be a sign of a larger problem. There are a large number of threads named ldlm_bl_38, and in fact most of the ldlm_bl_* threads have duplicate names. That in itself is harmless (though confusing while debugging), but may indicate that there is some race condition when starting up these threads that causes so many to be started in such a short time.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> ldlm_bl_thread_start(struct ldlm_bl_pool *blp) { struct ldlm_bl_thread_data bltd = { .bltd_blp = blp }; struct task_struct *task; init_completion(&amp;bltd.bltd_comp); bltd.bltd_num = atomic_read(&amp;blp-&gt;blp_num_threads); snprintf(bltd.bltd_name, sizeof(bltd.bltd_name) - 1, <span class="code-quote">"ldlm_bl_%02d"</span>, bltd.bltd_num); task = kthread_run(ldlm_bl_thread_main, &amp;bltd, bltd.bltd_name); <span class="code-keyword">if</span> (IS_ERR(task)) { CERROR(<span class="code-quote">"cannot start LDLM thread ldlm_bl_%02d: rc %ld\n"</span>, atomic_read(&amp;blp-&gt;blp_num_threads), PTR_ERR(task)); <span class="code-keyword">return</span> PTR_ERR(task); } wait_for_completion(&amp;bltd.bltd_comp); <span class="code-keyword">return</span> 0; } </pre> </div></div> <p>The naming problem is clearly in the code above - it picks the thread name before actually starting the thread and incrementing the counter.</p> <p>I counted about 200 threads started on this node, none of which were doing anything. It appears that the check for the number of running threads is also racy:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-comment">/* Not fatal <span class="code-keyword">if</span> racy and have a few too many threads */</span> <span class="code-keyword">static</span> <span class="code-object">int</span> ldlm_bl_thread_need_create(struct ldlm_bl_pool *blp, struct ldlm_bl_work_item *blwi) { <span class="code-object">int</span> busy = atomic_read(&amp;blp-&gt;blp_busy_threads); <span class="code-keyword">if</span> (busy &gt;= blp-&gt;blp_max_threads) <span class="code-keyword">return</span> 0; <span class="code-keyword">if</span> (busy &lt; atomic_read(&amp;blp-&gt;blp_num_threads)) <span class="code-keyword">return</span> 0; : : } </pre> </div></div> <p>and could be improved. I agree a small number of threads too many is not harmful, but it seems that this can go badly wrong in some cases.</p> <p>There is a module parameter that could be used to limit the number of threads, to add in <tt>/etc/modprobe.d/lustre.conf</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options ptlrpc ldlm_num_threads=16 </pre> </div></div> <p>which should reduce the number of threads significantly, but shouldn't cause any problems for clients since they don't handle many of these RPCs at one time. Note that this tunable may only fix the "symptom" (many LDLM threads) and not the root cause of the hang. It's hard to know for sure what is causing this problem since there are no error messages and the threads themselves are not doing anything either.</p> <blockquote> <p>After implementing the <tt>options ptlrpc ldlm_num_threads=16</tt> module option, the site has not experienced any further problems.</p></blockquote> <p>I think there are three items that need to be fixed here:</p> <ul class="alternate" type="square"> <li>figure out what is triggering so many ldlm_bl_&#42; threads being started.</li> <li>fix the thread naming problem. Creating the thread name before calling kthread_run() is no longer needed, it should be possible to pass the format string and thread index to kthread_run() directly, if it didn't cause a racy startup. It may be that we don't really want to have thread forking be <em>so</em> efficient</li> <li>limit the maximum number of <tt>ldlm_bl_&#42;</tt> threads on client namespaces to a lower number like 16 instead of 128, since clients don't handle many blocking callbacks compared to servers, and these numbers are per CPT.</li> </ul> <p>I scrutinized the code and looks the only possible reason of so many bl threads being started is the racy thread starting code itself.</p> <p>When there are lots of unused locks cached on client and kernel senses memory pressure, there could be lots bl work items being generated and processed by the bl threads, and if several bl threads running in to the ldlm_bl_thread_need_create() in parallel, lot more thread could be created mistakenly (it could exceed the blp_max_threads). Probably it caused the spinlock lockup problem as well? (too many threads contending on the blp_lock)</p> <p>I'll post a patch to fix the problem of forking too many threads exceeding limit (along with the wrong thread name) first. That could fix the spinlock lockup too.</p> <blockquote> <p>limit the maximum number of ldlm_bl_* threads on client namespaces to a lower number like 16 instead of 128, since clients don't handle many blocking callbacks compared to servers, and these numbers are per CPT.</p></blockquote> <p>Andreas, usually server sends blocking callbacks to client, so client should have more bl callbacks to handle, right? In addition, the bl threads on client needs to handle the unused lock cancel, so I don't think we should decrease the maximum thread number for client. (it's a per module parameter now, and bl threads are shared by all namespaces)</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17026" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17026</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7330" title="spinlock lockup on ldlm blp_lock in ldlm_bl_* threads" class="issue-link" data-issue-key="LU-7330"><del>LU-7330</del></a> ldlm: fix race of starting bl threads<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 187c285188130bb670c9f4daa38a67c35f064046</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17026/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17026/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7330" title="spinlock lockup on ldlm blp_lock in ldlm_bl_* threads" class="issue-link" data-issue-key="LU-7330"><del>LU-7330</del></a> ldlm: fix race of starting bl threads<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ebda41d8de7956f19fd27f86208c668e43c6957c</p> <p>Landed for 2.8</p> <p>Just to restate the workaround for this problem (so that it is more easily visible) is to add <tt>options ptlrpc ldlm_num_threads=16</tt> to <tt>/etc/modprobe.d/lustre.conf</tt> to avoid the problem until upgrading to a release in which this has been fixed.</p> Duplicate Related Development Rank 1|hzxr6v: Rank (Obsolete) 9223372036854775807 Severity