https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14713 Lustre <p>Write and Truncate IO will serialized on <tt>ll_trunc_sem::ll_trunc_{readers|waiters</tt>}, if one process quit abruptly (be killed), the other will keep waiting for the semaphore (task state be set as <tt>TASK_INTERRUPTIBLE</tt>):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> INFO: task a.out:109684 blocked for more than 120 seconds. Tainted: G IOE --------- - - 4.18.0-240.15.1.el8_3.x86_64 #1 "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Call Trace: __schedule+0x2a6/0x700 schedule+0x38/0xa0 trunc_sem_down_read+0xa6/0xb0 [lustre] vvp_io_write_start+0x107/0xb80 [lustre] cl_io_start+0x59/0x110 [obdclass] cl_io_loop+0x9a/0x1e0 [obdclass] ll_file_io_generic+0x380/0xb10 [lustre] ll_file_write_iter+0x136/0x5a0 [lustre] new_sync_write+0x124/0x170 vfs_write+0xa5/0x1a0 ksys_write+0x4f/0xb0 do_syscall_64+0x5b/0x1a0 </pre> </div></div> LU-14713

Process hung with waiting for mmap_sem

Bug Minor Resolved Fixed Zhenyu Xu Zhenyu Xu Thu, 27 May 2021 03:49:01 +0000 Wed, 8 Jun 2022 15:43:31 +0000 Sat, 20 Nov 2021 14:42:48 +0000 Lustre 2.12.7 Lustre 2.12.8 Lustre 2.15.0 0 11 <p>Making the wait interruptible is good. It would also be good if the interrupted process woke up any waiters for the lock, so that they do not wait forever.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/43844" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43844</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14713" title="Process hung with waiting for mmap_sem" class="issue-link" data-issue-key="LU-14713"><del>LU-14713</del></a> llite: make trunc_sem_down_xxx waiting killable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e335dad76a6edc7ede7a45b4ed60098dfa2c356f</p> <p>Thanks for the stack trace.</p> <p>I think there must have been a least one other process blocked on a mutex or semphore.  Was anything else in the logs?</p> <p>trunc_sem_down_read() can only block in two circumstances: 1/ ll_trunc_readers is negative so a write lock it held, 2/ ll_waiters is non-zero so some process is waiting for a write lock.</p> <p>In case 2, that process will be blocked on trunc_sem_down_write() and should get a soft-lockup warning.</p> <p>In case 1, the writer must have called vvp_io_setattr_start and either</p> <p>A/ not yet called vvp_io_setattr_end(), or</p> <p>B/ changed io-&gt;ci_type or cio-&gt;u.ci_setattr.sa_subtype so taht when vvp_io_setattr_end() was called it didn't release the lock.</p> <p>B seems extremely unlikely.</p> <p>A implies that the process is still waiting as _end is <b>always</b> called after _start before the process can exit or return to user space.</p> <p>The only scenario that I can think of (other than 1B) where there would be no other process triggering the soft-lockup detector is that the process holding the write lock is in an TASK_INTERRUPTIBLE or TASK_IDLE wait.  That seems unlikely, but should be easy to find if it is the case.</p> <p>Given that the program that generated the stack trace was "a.out" I assume you can reproduce the problem.  Could you do that, then "echo t &gt; /proc/sysrq-trigger" to generate a stack trace of all tasks.  Then provide me with the full set of stack traces?</p> <p> </p> <p>I agree that it would be good to be able to kill tasks blocked on mutexes or semphores, but that tends to take a lot more careful effort than just returning an error code.</p> <p> </p> <p>I attached stack traces in DDN-2177. Please note that this issue occurred twice on this client and the process "a.out:109684" in problem description is when this occurred first time. The customer restarted the client, the issue occurred again and then vmcore was captured (vmcore is available in DDN-2177). The stack traces attached is from this vmcore and so the process is "a.out:4718". Hope this helps, </p> <p>I don't know how to access "DDN-2177".  I don't suppose you can attach them here?</p> <p> </p> <p>Ah, we usually don't share our customer's data publicly. So I attached it to DDN-2177, but it would be better to digging into the root cause. <br/> Here is the stack trace: <span class="error">&#91;^crash.log&#93;</span> </p> <p>I may delete the attachment later. </p> <p>Thanks for the crash data.  I could not find the other process that I was sure must exist.</p> <p>The only process that looks at all suspicious is PID 46556 (ll_imp_inval) and I suspect it is waiting on the a.out thread, not the other way around.</p> <p>It is very likely that the ll_trunc_sem that a.out is blocked on has ll_trunc_waiters == 0 and ll_trunc_reads == -1.</p> <p>There are two ways this can happen:</p> <p>1/ a thread calls trunc_sem_down_write, but never calls trunc_sem_up_write</p> <p>2/ a thread calls trunc_sem_up_write without having first called trunc_sem_down_write.  If it does this while some other thread is holding a read lock, then the up_write call will set ll_trunc_reads to 0, and when the other thread calls trunc_sem_up_read, ll_trunc_reads will be decremented to -1.</p> <p>I cannot see a code path that leads to option 1.  I <b>can</b> see a code path that might lead to option 2, but there may be considerations that I'm not aware of which make it impossible.  I'll explain it in case it is possible.</p> <p>trunc_sem_up/down_write are only called from vvp_io_setattr_start/end.  These are only called from cl_io_start/end which are always paired.</p> <p>They are called directly in ll_readahead_handle_work and cl_io_loop where cl_io_end will <b>always</b> follow cl_io_start, and indirectly through love_io_start/end which will be paired properly exactly if  vvp_io_setattr_start/end are paired properly.</p> <p>cl_io_end will <b>always</b> call all of the -&gt;cio_end functions, so missing a call to trunc_sem_up write is (should be) impossible.</p> <p>However cl_io_start will <b>sometimes</b> <b>not</b>  call all cio_start functions.  If one of the cio_start functions in the stack returns non-zero, the remainder are not called.  However the matching cio_end function will be called.  This seems <b>very</b> <b>strange</b>.  I might be missing something, but it certainly seems that is the what would happen.</p> <p>So if some cio_start function before vvp_io_setattr_start returned non-zero, then trunc_sem_down_write would not be called, but trunc_semd_up_write would.  This would often not cause a problem.  But if some other thread had called trunc_sem_down_read(), then it would provide exactly the symptom you observe.</p> <p>@Andreas does this seem at all possible?</p> <p> </p> <blockquote> <p>@Andreas does this seem at all possible?</p></blockquote> <p>That is really a question for Bobijam to answer, as I'm not nearly as expert in the client IO code as he is.</p> <p>The clio layer always (at least for now) stacks like this VVP upon LOV upon OSC/MDC, and cl_io_start() calls from top to bottom, which is vvp_io_xxx_start() then lov_io_xxx_start() which is nested with {mdc|osc}_io_xxx_start(), and that makes vvp_io_setattr_start() the first to be called in the CIT_SETATTR cl_io_loop call path.</p> <p>Like following example shows</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00000001:0.0:1622711194.323939:0:20024:0:(cl_io.c:755:cl_io_loop()) Process entered 00000020:00000001:0.0:1622711194.323940:0:20024:0:(cl_io.c:390:cl_io_iter_init()) Process entered ... 00000020:00000001:0.0:1622711194.323955:0:20024:0:(cl_io.c:402:cl_io_iter_init()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1622711194.323958:0:20024:0:(cl_io.c:311:cl_io_lock()) Process entered ... 00000020:00000001:0.0:1622711194.324421:0:20024:0:(cl_io.c:331:cl_io_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1622711194.324422:0:20024:0:(cl_io.c:516:cl_io_start()) Process entered 00000080:00000001:0.0:1622711194.324422:0:20024:0:(vvp_io.c:737:vvp_io_setattr_start()) Process entered ... 00000080:00000001:0.0:1622711194.324427:0:20024:0:(vvp_io.c:763:vvp_io_setattr_start()) Process leaving (rc=0 : 0 : 0) 00020000:00000001:0.0:1622711194.324427:0:20024:0:(lov_io.c:1487:lov_io_setattr_start()) Process entered ... 00000008:00000001:0.0:1622711194.324429:0:20024:0:(osc_io.c:582:osc_io_setattr_start()) Process entered ... 00000008:00000001:0.0:1622711194.324459:0:20024:0:(osc_io.c:687:osc_io_setattr_start()) Process leaving (rc=0 : 0 : 0) ... 00020000:00000001:0.0:1622711194.324461:0:20024:0:(lov_io.c:1502:lov_io_setattr_start()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1622711194.324462:0:20024:0:(cl_io.c:528:cl_io_start()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>So basically vvp_io_setattr_start() will be called whatever in CIT_SETATTR cl_io_start().</p> <p>Thanks for the excellent explanation.  The bug must be hiding somewhere else then.  I'll keep digging.</p> <p> </p> <p>Could you please report precisely which version of Lustre is in used (ideally the git commit hash).  The "Affected versions" above says "None" which isn't helpful.</p> <p> </p> <p>Another question:  If you still have the core file and can find the 'struct ll_trunc_sem"  that the process is blocked on, could you report the contents of that.  I'm guessing the two atomic_t's are -1 and 0, but it would be highly informative if they had other values.  Maybe some sort of memory corruption caused the problem.</p> <p> </p> <p>The reported version is 2.12.6-ddn19 (both client and server). And unfortunately I failed to load teh vmcore with crqsh</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># crash ./120035_YITP_client_ioerror/usr/lib/debug/lib/modules/4.18.0-240.15.1.el8_3.x86_64/vmlinux 120035_YITP_client_ioerror/vmcore  crash 7.2.3-11.el7_9.1 Copyright (C) 2002-2017  Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation Copyright (C) 1999-2006  Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited Copyright (C) 2006, 2007  VA Linux Systems Japan K.K. Copyright (C) 2005, 2011  NEC Corporation Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions.  Enter "help copying" to see the conditions. This program has absolutely no warranty.  Enter "help warranty" for details.   GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt; This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.  Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [222MB]: patching 97897 gdb minimal_symbol values please wait... (gathering task table data)                              crash: radix trees do not exist or have changed their format </pre> </div></div> <p>&gt; The reported version is 2.12.6-ddn19</p> <p>The public "lustre-release" git tree doesn't have any branch or tag with a name like that, and the public "2.12.6" doesn't contain trunc_sem_down_read() at all.</p> <p>Is there somewhere I can get the source code for that release?</p> <blockquote><p>I can open it, but I'm having difficulty to find 'struct ll_trunc_sem'. vmcore is available from:</p></blockquote> <p>same here.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; struct ll_trunc_sem struct: invalid data structure reference: ll_trunc_sem</pre> </div></div> <p>seems lack of lustre module debug info.</p> <p>rpm -qpl lustre-client-debuginfo-2.12.6_ddn19-1.el8.x86_64.rpm</p> <p>shows that it does not contain lustre module info, only some tools.</p> <p>Looking at the vmcore  "bt -FF 4718" show</p> <p> </p> <p>#1 <span class="error">&#91;ffffbe8540f3bb88&#93;</span> schedule at ffffffff8f6d38a8<br/> ffffbe8540f3bb90: <span class="error">&#91;ffff9f9b3d8c37a0:lustre_inode_cache&#93;</span> trunc_sem_down_read+166 <br/> #2 <span class="error">&#91;ffffbe8540f3bb98&#93;</span> trunc_sem_down_read at ffffffffc12523c6 <span class="error">&#91;lustre&#93;</span><br/> ffffbe8540f3bba0: <span class="error">&#91;ffff9f9b3d8c37a0:lustre_inode_cache&#93;</span> 00000000ffffffff</p> <p> </p> <p>So ffffbe8540f3bba0 must be the address of the ll_trunc_sem.</p> <p>crash&gt; x/2wx 0xffff9f9b3d8c37a0<br/> 0xffff9f9b3d8c37a0: 0x00000000 0x00000000</p> <p>so both ll_trunc_waiters and ll_trunc_readers are zero.  So trunc_sem_down_read() shouldn't block.</p> <p>This suggests a missed wake-up.  It could only be the wakeup from trunc_sem_up_write().  Maybe a barrier is needed after the atomic_set, and before the atomic_add_unless_negative.</p> <p>But I thought barriers like that weren't needed on x86.</p> <p>I'll read up about memory ordering again.</p> <p> </p> <p>I'm confident that memory barriers aren't a problem.  atomic_set() and atomic_read() provide any memory barriers they need, and none are needed on x86.</p> <p>So I'm stumped - for today at least.</p> <p>ll_trunc_readers is definitely 0 and whenever it is set to zero a wake_up happens.  But the waiter didn't wake up.</p> <p>I'd still like to see the exact lustre source code for these modules, but I doubt it will show anything.</p> <p> </p> <p>Neil,</p> <p>I'm not familiar with the memory barrier while I noticed a kernel comment about wake_up_var()&#45;&gt;__wake_up_bit()&#45;&gt;waitqueue_active()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 97 /** 98 * waitqueue_active -- locklessly test for waiters on the queue 99 * @wq_head: the waitqueue to test for waiters 100 * 101 * returns true if the wait list is not empty 102 * 103 * NOTE: this function is lockless and requires care, incorrect usage _will_ 104 * lead to sporadic and non-obvious failure. 105 * 106 * Use either while holding wait_queue_head::lock or when used for wakeups 107 * with an extra smp_mb() like:: 108 * 109 * CPU0 - waker CPU1 - waiter 110 * 111 * for (;;) { 112 * @cond = true; prepare_to_wait(&amp;wq_head, &amp;wait, state); 113 * smp_mb(); // smp_mb() from set_current_state() 114 * if (waitqueue_active(wq_head)) if (@cond) 115 * wake_up(wq_head); break; 116 * schedule(); 117 * } 118 * finish_wait(&amp;wq_head, &amp;wait); 119 * 120 * Because without the explicit smp_mb() it's possible for the 121 * waitqueue_active() load to get hoisted over the @cond store such that we'll 122 * observe an empty wait list while the waiter might not observe @cond. </pre> </div></div> <p>Does that mean there should be smp_mb() in trunc_sem_up_write() before wake_up_var()?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 347 <span class="code-keyword">static</span> inline void trunc_sem_up_write(struct ll_trunc_sem *sem) 348 { 349 atomic_set(&amp;sem-&gt;ll_trunc_readers, 0); 350 wake_up_var(&amp;sem-&gt;ll_trunc_readers); 351 } </pre> </div></div> <p>It seems I was wrong about atomic_set implying a barrier, but I'm slightly less convinced that x86 doesn't require them.</p> <p>So I'd still be very surprised if barriers were causing the problem, but there is something worth fixing there.</p> <p> </p> <p>The atomic_set in trunc_sem_uip_write should be atomic_set_release(), and  that atomic_reads should be atomic_read_acquire().</p> <p> </p> <p>I would still like to see the exact source code for the kernel modules which produced the hang, if that is possible.</p> <p> </p> <p>Neil Brown (neilb@suse.de) uploaded a new patch: <a href="https://review.whamcloud.com/44372" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44372</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14713" title="Process hung with waiting for mmap_sem" class="issue-link" data-issue-key="LU-14713"><del>LU-14713</del></a> llite: add memory barriier to the trunc_sem.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1b20297bce772534dd557abaa11859447da8651b</p> <p>"Bobi Jam &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/44715" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44715</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14713" title="Process hung with waiting for mmap_sem" class="issue-link" data-issue-key="LU-14713"><del>LU-14713</del></a> llite: tighten condition for fault not drop mmap_sem<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0cccc61c4bd48cc6942963394cd2ea70e30b8563</p> <p>As ll_fault0() shows, if ll_filemap_fault() returns VM_FAULT_RETRY, it presumes mmap_sem has not dropped and continues to normal fault under DLM lock, which will eventually drop the mmap_sem.</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>ll_fault0()</b></div><div class="codeContent panelContent"> <pre class="code-java"> 279 <span class="code-keyword">if</span> (ll_sbi_has_fast_read(ll_i2sbi(file_inode(vma-&gt;vm_file)))) { 280 <span class="code-comment">/* <span class="code-keyword">do</span> fast fault */</span> 281 bool has_retry = vmf-&gt;flags &amp; FAULT_FLAG_RETRY_NOWAIT; 282 283 <span class="code-comment">/* To avoid loops, instruct downstream to not drop mmap_sem */</span> 284 vmf-&gt;flags |= FAULT_FLAG_RETRY_NOWAIT; 285 ll_cl_add(vma-&gt;vm_file, env, NULL, LCC_MMAP); 286 fault_ret = ll_filemap_fault(vma, vmf); 287 ll_cl_remove(vma-&gt;vm_file, env); 288 <span class="code-keyword">if</span> (!has_retry) 289 vmf-&gt;flags &amp;= ~FAULT_FLAG_RETRY_NOWAIT; 290 291 /* - If there is no error, then the page was found in cache and 292 * uptodate; 293 * - If VM_FAULT_RETRY is set, the page existed but failed to 294 * lock. We will <span class="code-keyword">try</span> slow path to avoid loops. 295 * - Otherwise, it should <span class="code-keyword">try</span> normal fault under DLM lock. */ 296 <span class="code-keyword">if</span> (!(fault_ret &amp; VM_FAULT_RETRY) &amp;&amp; 297 !(fault_ret &amp; VM_FAULT_ERROR)) 298 GOTO(out, result = 0); 299 300 fault_ret = 0; 301 } </pre> </div></div> <p>But filemap_fault() shows that returning VM_FAULT_RETRY while not dopping mmap_sem needs vmf-&gt;flags contains both FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT, otherwise it could possibly return VM_FAULT_RETRY with mmap_sem released.</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>filemap_fault()</b></div><div class="codeContent panelContent"> <pre class="code-java"> 2421 <span class="code-keyword">if</span> (!lock_page_or_retry(page, vma-&gt;vm_mm, vmf-&gt;flags)) { 2422 page_cache_release(page); 2423 <span class="code-keyword">return</span> ret | VM_FAULT_RETRY; 2424 } </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 410 <span class="code-keyword">static</span> inline <span class="code-object">int</span> lock_page_or_retry(struct page *page, struct mm_struct *mm, 411 unsigned <span class="code-object">int</span> flags) 412 { 413 might_sleep(); 414 <span class="code-keyword">return</span> trylock_page(page) || __lock_page_or_retry(page, mm, flags); 415 } </pre> </div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 854 <span class="code-object">int</span> __lock_page_or_retry(struct page *page, struct mm_struct *mm, 855 unsigned <span class="code-object">int</span> flags) 856 { 857 <span class="code-keyword">if</span> (flags &amp; FAULT_FLAG_ALLOW_RETRY) { 858 /* 859 * CAUTION! In <span class="code-keyword">this</span> <span class="code-keyword">case</span>, mmap_sem is not released 860 * even though <span class="code-keyword">return</span> 0. 861 */ 862 <span class="code-keyword">if</span> (flags &amp; FAULT_FLAG_RETRY_NOWAIT) 863 <span class="code-keyword">return</span> 0; 864 865 up_read(&amp;mm-&gt;mmap_sem); 866 <span class="code-keyword">if</span> (flags &amp; FAULT_FLAG_KILLABLE) 867 wait_on_page_locked_killable(page); 868 <span class="code-keyword">else</span> 869 wait_on_page_locked(page); 870 <span class="code-keyword">return</span> 0; 871 } <span class="code-keyword">else</span> { 872 <span class="code-keyword">if</span> (flags &amp; FAULT_FLAG_KILLABLE) { 873 <span class="code-object">int</span> ret; 874 875 ret = __lock_page_killable(page); 876 <span class="code-keyword">if</span> (ret) { 877 up_read(&amp;mm-&gt;mmap_sem); 878 <span class="code-keyword">return</span> 0; 879 } 880 } <span class="code-keyword">else</span> 881 __lock_page(page); 882 <span class="code-keyword">return</span> 1; 883 } 884 } </pre> </div></div> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44715/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44715/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14713" title="Process hung with waiting for mmap_sem" class="issue-link" data-issue-key="LU-14713"><del>LU-14713</del></a> llite: tighten condition for fault not drop mmap_sem<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 81aec05103558f57adb10fd319847304cdf44aa7</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/43844/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43844/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14713" title="Process hung with waiting for mmap_sem" class="issue-link" data-issue-key="LU-14713"><del>LU-14713</del></a> llite: mend the trunc_sem_up_write()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 39745c8b5493159bbca62add54ca9be7cac6564f</p> <p>Landed for 2.15</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47404" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47404</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14713" title="Process hung with waiting for mmap_sem" class="issue-link" data-issue-key="LU-14713"><del>LU-14713</del></a> llite: tighten condition for fault not drop mmap_sem<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: cdf66f48b5100b0b209be3f1741e6163000299e2</p> <p>We likely hit this issue at the CEA in production.<br/> After upgrading from 2.12.6 to 2.12.7, we observed regularly hangs on the clients when canceling a job.<br/> The hang is observed on mmap_sem when closing the /dev/infiniband/uverb* device:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>void ib_umem_release(struct ib_umem *umem) { if (!umem) return; ..... #ifdef HAVE_ATOMIC_PINNED_VM atomic64_sub(ib_umem_num_pages(umem), &amp;umem-&gt;owning_mm-&gt;pinned_vm); #else down_write(&amp;umem-&gt;owning_mm-&gt;mmap_sem); &lt;----- the thread hang here, rw_semaphore owner is garbage (already free) </pre> </div></div> <p>MOFED: 5.4<br/> kernel: 3.10<br/> lustre: client 2.12.7</p> <p>So I am not 100% sure, but I think we hit the regression from 21dc165991 "<a href="https://jira.whamcloud.com/browse/LU-13182" title="MAP_POPULATE hangs with Linux 5.4" class="issue-link" data-issue-key="LU-13182"><del>LU-13182</del></a> llite: Avoid eternel retry loops with MAP_POPULATE" (landed in 2.12.7).<br/> This patch could cause mmap_sem to be released (up_read) twice (only for kernel &lt; 5.1).</p> Duplicate Related LU-15397 Development Rank 1|i01vgn: Rank (Obsolete) 9223372036854775807 Severity