https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13908 Lustre <p>On compute node, ASSERT fails and node crashes. One node reports two failed ASSERTs in the dumped log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 13759:0:(ldlm_lock.c:213:ldlm_lock_put()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed: LustreError: 10188:0:(ldlm_lock.c:205:ldlm_lock_put()) ASSERTION( atomic_read(&amp;lock-&gt;l_refc) &gt; 0 ) failed: LustreError: 10188:0:(ldlm_lock.c:205:ldlm_lock_put()) LBUG Pid: 10188, comm: ldlm_bl_16 3.10.0-1127.0.0.1chaos.ch6.x86_64 #1 SMP Fri Apr 3 08:56:52 PDT 2020 Call Trace: [&lt;ffffffffc0a637ec&gt;] libcfs_call_trace+0x8c/0xd0 [libcfs] [&lt;ffffffffc0a638ac&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [&lt;ffffffffc16cb366&gt;] ldlm_lock_put+0x616/0x7b0 [ptlrpc] [&lt;ffffffffc0c5828b&gt;] osc_extent_put+0x6b/0x320 [osc] [&lt;ffffffffc0c645fb&gt;] osc_cache_wait_range+0x30b/0x960 [osc] [&lt;ffffffffc0c655ce&gt;] osc_cache_writeback_range+0x97e/0x1000 [osc] [&lt;ffffffffc0c51195&gt;] osc_lock_flush+0x195/0x290 [osc] [&lt;ffffffffc0c51653&gt;] osc_ldlm_blocking_ast+0x2e3/0x3a0 [osc] [&lt;ffffffffc16d2dea&gt;] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] [&lt;ffffffffc16ea620&gt;] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] [&lt;ffffffffc16f03f7&gt;] ldlm_cli_cancel+0x157/0x620 [ptlrpc] [&lt;ffffffffc0c514ea&gt;] osc_ldlm_blocking_ast+0x17a/0x3a0 [osc] [&lt;ffffffffc16fc618&gt;] ldlm_handle_bl_callback+0xf8/0x4f0 [ptlrpc] [&lt;ffffffffc16fd230&gt;] ldlm_bl_thread_main+0x820/0xa60 [ptlrpc] [&lt;ffffffffbaccca01&gt;] kthread+0xd1/0xe0 [&lt;ffffffffbb3bff5d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff Kernel panic - not syncing: LBUG CPU: 53 PID: 10188 Comm: ldlm_bl_16 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1127.0.0.1chaos.ch6.x86_64 #1 Hardware name: Penguin Computing Relion OCP1930e/S2600KPR, BIOS SE5C610.86B.01.01.0027.071020182329 07/10/2018 </pre> </div></div> <p>The other reports the same ASSERT twice:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 20571:0:(ldlm_lock.c:213:ldlm_lock_put()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed: LustreError: 36887:0:(ldlm_lock.c:213:ldlm_lock_put()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed: LustreError: 36887:0:(ldlm_lock.c:213:ldlm_lock_put()) LBUG Pid: 36887, comm: ldlm_bl_62 3.10.0-1127.0.0.1chaos.ch6.x86_64 #1 SMP Fri Apr 3 08:56:52 PDT 2020 Call Trace: [&lt;ffffffffc0a727ec&gt;] libcfs_call_trace+0x8c/0xd0 [libcfs] [&lt;ffffffffc0a728ac&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [&lt;ffffffffc176f3ca&gt;] ldlm_lock_put+0x67a/0x7b0 [ptlrpc] [&lt;ffffffffc1773058&gt;] ldlm_lock_match_with_skip+0x3b8/0x860 [ptlrpc] [&lt;ffffffffc0d982d2&gt;] osc_match_base+0x102/0x290 [osc] [&lt;ffffffffc0da3dfc&gt;] osc_obj_dlmlock_at_pgoff+0x14c/0x2c0 [osc] [&lt;ffffffffc0d9c358&gt;] osc_req_attr_set+0x128/0x610 [osc] [&lt;ffffffffc1549b13&gt;] cl_req_attr_set+0x63/0x160 [obdclass] [&lt;ffffffffc0d969f3&gt;] osc_build_rpc+0x483/0x1080 [osc] [&lt;ffffffffc0db1cbd&gt;] osc_io_unplug0+0xecd/0x19c0 [osc] [&lt;ffffffffc0db6620&gt;] osc_cache_writeback_range+0x9d0/0x1000 [osc] [&lt;ffffffffc0da2195&gt;] osc_lock_flush+0x195/0x290 [osc] [&lt;ffffffffc0da2653&gt;] osc_ldlm_blocking_ast+0x2e3/0x3a0 [osc] [&lt;ffffffffc1776dea&gt;] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] [&lt;ffffffffc178e620&gt;] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] [&lt;ffffffffc17943f7&gt;] ldlm_cli_cancel+0x157/0x620 [ptlrpc] [&lt;ffffffffc0da24ea&gt;] osc_ldlm_blocking_ast+0x17a/0x3a0 [osc] [&lt;ffffffffc17a0618&gt;] ldlm_handle_bl_callback+0xf8/0x4f0 [ptlrpc] [&lt;ffffffffc17a1230&gt;] ldlm_bl_thread_main+0x820/0xa60 [ptlrpc] [&lt;ffffffffab4cca01&gt;] kthread+0xd1/0xe0 [&lt;ffffffffabbbff5d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff Kernel panic - not syncing: LBUG CPU: 20 PID: 36887 Comm: ldlm_bl_62 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1127.0.0.1chaos.ch6.x86_64 #1</pre> </div></div> <p>From /tftpboot/dumps/192.168.64.82-2020-08-12-13:23:27/vmcore-dmesg.txt<br/> and /tftpboot/dumps/192.168.66.180-2020-08-12-16:39:36/vmcore-dmesg.txt</p> 2.12.4_5.chaos <br/> toss 3.6-2 (RHEL 7.8) LU-13908

ldlm_lock_put()) ASSERTION( (((( lock))->l_flags & (1ULL << 50)) != 0) ) failed

Bug Critical Resolved Fixed Zhenyu Xu Olaf Faaland llnl Mon, 17 Aug 2020 22:20:37 +0000 Mon, 25 Jan 2021 17:37:39 +0000 Mon, 25 Jan 2021 17:37:39 +0000 Lustre 2.12.4 1 7 <p>Looks like dupe of <a href="https://jira.whamcloud.com/browse/LU-13089" title="ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed" class="issue-link" data-issue-key="LU-13089"><del>LU-13089</del></a></p> <p>For my reference, my local ticket is TOSS4864</p> <p>Average about 10 crashes per week, although it varies widely. I do not know whether a particular workload triggers it.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ofaaland" class="user-hover" rel="ofaaland">ofaaland</a> has this issue ever been seen on earlier version of 2.12.x or or 2.10.x?</p> <p>Bobijam</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>I agree that this is likely a duplicate of <a href="https://jira.whamcloud.com/browse/LU-13089" title="ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed" class="issue-link" data-issue-key="LU-13089"><del>LU-13089</del></a>. As Oleg notes there "except this time it's glimpse cb vs cancel cb race". Based on the functions changed and the time that this was first noticed I suspect that this was introduced by</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 2548cb9e32bfca897de577f88836629f72641369 Author: Patrick Farrell &lt;pfarrell@whamcloud.com&gt; AuthorDate: Mon Sep 9 11:56:07 2019 -0400 Commit: Oleg Drokin &lt;green@whamcloud.com&gt; CommitDate: Thu Dec 12 23:05:15 2019 +0000 LU-11670 osc: glimpse - search for active lock </pre> </div></div> <p>Lustre-change: <a href="https://review.whamcloud.com/33660" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33660</a><br/> Reviewed-on: <a href="https://review.whamcloud.com/36406" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36406</a></p> <p>Peter,<br/> We have never seen this under an earlier version of Lustre 2.12.x, but this is the first 2.12 we deployed widely.<br/> We have never seen this under any 2.10.x version.<br/> thanks</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39693" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39693</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13908" title="ldlm_lock_put()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed" class="issue-link" data-issue-key="LU-13908"><del>LU-13908</del></a> osc: revert "glimpse - search for active lock"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6caffe3423c184b9717c9a5307da503ae0fbba4e</p> <p>The commit mentioned looks like the culprit. Olaf, is it possible trying the revert patch?</p> <p>Yes, I can try.  It may take a while.  Sorry for the delay answering your question.</p> <p>Hi Zhenyu,<br/> Can you get the revert patch to pass Maloo, so I can more confidently test this on the production system where we see the error? We have not been able to reproduce this on our test systems.<br/> thanks</p> <p>I just noticed that Gerrit did not post an update to Jira for the revert on b2_12 - <a href="https://review.whamcloud.com/#/c/39819/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/39819/</a> . Hopefully this is more convenient to test <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ofaaland" class="user-hover" rel="ofaaland">ofaaland</a></p> <p>For my tracking purposes: my internal ticket is TOSS4864</p> <p>Hello, </p> <p>Do you have any feedback on whether the revert of the targeted commit has any effect on your production ?</p> <p>One of our customer has hit this 8 times in the past week. </p> <p>This would not be a problem if it affected only the crash client, but in our case, the oss handling the lock never releases it until the crashed client remounts the filesystem (the oss keeps retrying to send requests to the crashed client every 600 seconds even hours after the crash), which sometimes will result in a almost hung filesystem.</p> <p>I am surprised the client is never evicted, would there be a reason to this ?</p> <p>This is believed to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-11719" title="Refactor search_itree" class="issue-link" data-issue-key="LU-11719"><del>LU-11719</del></a></p> <p>Next week we will finally get a build with the revert installed on the machine where we've seen the issue.</p> <p>Wouldn't you rather test the actual fix?</p> <p>Yes, but it's sadly complicated.</p> <p>In about 3 weeks we will have 2.12.6_3.llnl, which includes the fix from <a href="https://jira.whamcloud.com/browse/LU-11719" title="Refactor search_itree" class="issue-link" data-issue-key="LU-11719"><del>LU-11719</del></a>, on the clusters where we've seen this.</p> Related LU-13089 LU-11719 Development Rank 1|i017xj: Rank (Obsolete) 9223372036854775807 Severity