https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9563 Lustre <p>System was running normally, no fault was induced. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[25789.588202] LustreError: 4166:0:(ldlm_lock.c:2548:ldlm_lock_downgrade()) ASSERTION( lock-&gt;l_granted_mode &amp; (LCK_PW | LCK_EX) ) failed: ^M [25789.609363] LustreError: 4166:0:(ldlm_lock.c:2548:ldlm_lock_downgrade()) LBUG^M [25789.621115] Pid: 4166, comm: ptlrpc_hr01_005^M [25789.629112] ^M [25789.629112] Call Trace:^M [25789.639919] [&lt;ffffffffa084d7ee&gt;] libcfs_call_trace+0x4e/0x60 [libcfs]^M [25789.650059] [&lt;ffffffffa084d87c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs]^M [25789.659583] [&lt;ffffffffa0b5883b&gt;] ldlm_lock_downgrade+0x19b/0x1d0 [ptlrpc]^M [25789.669915] [&lt;ffffffffa0bae96f&gt;] ptlrpc_hr_main+0x5bf/0x910 [ptlrpc]^M [25789.679492] [&lt;ffffffff810c8345&gt;] ? sched_clock_cpu+0x85/0xc0^M [25789.688107] [&lt;ffffffff810c54c0&gt;] ? default_wake_function+0x0/0x20^M [25789.697330] [&lt;ffffffffa0bae3b0&gt;] ? ptlrpc_hr_main+0x0/0x910 [ptlrpc]^M [25789.706522] [&lt;ffffffff810b0a4f&gt;] kthread+0xcf/0xe0^M [25789.714056] [&lt;ffffffff810b0980&gt;] ? kthread+0x0/0xe0^M [25789.721469] [&lt;ffffffff81697318&gt;] ret_from_fork+0x58/0x90^M [25789.729355] [&lt;ffffffff810b0980&gt;] ? kthread+0x0/0xe0^M [25789.736613] ^M [25789.740048] Kernel panic - not syncing: LBUG^M </pre> </div></div> <p>System wedged hard at this time. Will reboot and run with full debug</p> soak performance cluster LU-9563

LBUG ldlm_lock_downgrade()) ASSERTION( lock->l_granted_mode & (LCK_PW | LCK_EX) ) failed:

Bug Blocker Resolved Cannot Reproduce Lai Siyao Cliff White soak Fri, 26 May 2017 14:11:38 +0000 Mon, 5 Jun 2017 17:39:56 +0000 Mon, 5 Jun 2017 17:39:56 +0000 Lustre 2.10.0 0 6 <p>Soak was running the build described at <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Soak#SoakTestingonSoak-20170525" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Soak#SoakTestingonSoak-20170525</a></p> <p>Lai</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p><a href="https://review.whamcloud.com/#/c/27207/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/27207/</a> for <a href="https://jira.whamcloud.com/browse/LU-9504" title="LBUG ptlrpc_handle_rs()) ASSERTION( lock != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-9504"><del>LU-9504</del></a> doesn't consider race with mdt_steal_ack_lock(), I'll update that patch.</p> <p>ok then as <a href="https://jira.whamcloud.com/browse/LU-9504" title="LBUG ptlrpc_handle_rs()) ASSERTION( lock != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-9504"><del>LU-9504</del></a> is not landed yet, let's close this asa duplicate of <a href="https://jira.whamcloud.com/browse/LU-9504" title="LBUG ptlrpc_handle_rs()) ASSERTION( lock != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-9504"><del>LU-9504</del></a></p> <p>Soak is dead until this issue is fixed.</p> <p>Tested with latest patch of <a href="https://jira.whamcloud.com/browse/LU-9504" title="LBUG ptlrpc_handle_rs()) ASSERTION( lock != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-9504"><del>LU-9504</del></a><br/> <a href="https://review.whamcloud.com/#/c/27207/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/27207/</a>'</p> <p>Soak hit this issue immediately.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">ay 31 17:11:18 soak-9 kernel: LustreError: 4177:0:(ldlm_lock.c:2548:ldlm_lock_downgrade()) ASSERTION( lock-&gt;l_granted_mode &amp; (LCK_PW | LCK_EX) ) failed: May 31 17:11:18 soak-9 kernel: LustreError: 4177:0:(ldlm_lock.c:2548:ldlm_lock_downgrade()) LBUG May 31 17:11:18 soak-9 kernel: Pid: 4177, comm: ptlrpc_hr01_003 May 31 17:11:18 soak-9 kernel: #012Call Trace: May 31 17:11:18 soak-9 kernel: [&lt;ffffffffa08247ee&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] May 31 17:11:19 soak-9 kernel: [&lt;ffffffffa082487c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] May 31 17:11:19 soak-9 kernel: [&lt;ffffffffa0b6283b&gt;] ldlm_lock_downgrade+0x19b/0x1d0 [ptlrpc] May 31 17:11:19 soak-9 kernel: [&lt;ffffffffa0bb9620&gt;] ptlrpc_handle_rs+0x3f0/0x640 [ptlrpc] May 31 17:11:19 soak-9 kernel: [&lt;ffffffffa0bb9955&gt;] ptlrpc_hr_main+0xe5/0x2c0 [ptlrpc] May 31 17:11:19 soak-9 kernel: [&lt;ffffffff810c54c0&gt;] ? default_wake_function+0x0/0x20 May 31 17:11:19 soak-9 kernel: [&lt;ffffffffa0bb9870&gt;] ? ptlrpc_hr_main+0x0/0x2c0 [ptlrpc] May 31 17:11:19 soak-9 kernel: [&lt;ffffffff810b0a4f&gt;] kthread+0xcf/0xe0 May 31 17:11:19 soak-9 kernel: [&lt;ffffffff810b0980&gt;] ? kthread+0x0/0xe0 May 31 17:11:19 soak-9 kernel: [&lt;ffffffff81697318&gt;] ret_from_fork+0x58/0x90 May 31 17:11:19 soak-9 kernel: [&lt;ffffffff810b0980&gt;] ? kthread+0x0/0xe0 May 31 17:11:19 soak-9 kernel: May 31 17:11:19 soak-9 kernel: Kernel panic - not syncing: LBUG </pre> </div></div> <p>Soak is dead until we see a fix for this.</p> <p>Attempted to test latest patch for <a href="https://jira.whamcloud.com/browse/LU-9504" title="LBUG ptlrpc_handle_rs()) ASSERTION( lock != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-9504"><del>LU-9504</del></a>, hit this issue again immediately.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jun 5 16:51:12 soak-11 kernel: LustreError: 4204:0:(ldlm_lock.c:2548:ldlm_lock_downgrade()) ASSERTION( lock-&gt;l_granted_mode &amp; (LCK_PW | LCK_EX) ) failed: Jun 5 16:51:12 soak-11 kernel: LustreError: 4204:0:(ldlm_lock.c:2548:ldlm_lock_downgrade()) LBUG Jun 5 16:51:12 soak-11 kernel: Pid: 4204, comm: ptlrpc_hr01_002 Jun 5 16:51:12 soak-11 kernel: #012Call Trace: Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffffa08637ee&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffffa086387c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffffa0b6e83b&gt;] ldlm_lock_downgrade+0x19b/0x1d0 [ptlrpc] Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffffa0bc57f8&gt;] ptlrpc_handle_rs+0x5c8/0x700 [ptlrpc] Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffffa0bc5a15&gt;] ptlrpc_hr_main+0xe5/0x2c0 [ptlrpc] Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffff810c54c0&gt;] ? default_wake_function+0x0/0x20 Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffffa0bc5930&gt;] ? ptlrpc_hr_main+0x0/0x2c0 [ptlrpc] Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffff810b0a4f&gt;] kthread+0xcf/0xe0 Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffff810b0980&gt;] ? kthread+0x0/0xe0 Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffff81697318&gt;] ret_from_fork+0x58/0x90 Jun 5 16:51:12 soak-11 kernel: [&lt;ffffffff810b0980&gt;] ? kthread+0x0/0xe0 </pre> </div></div> <p>Still hitting this issue, with every new <a href="https://jira.whamcloud.com/browse/LU-9504" title="LBUG ptlrpc_handle_rs()) ASSERTION( lock != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-9504"><del>LU-9504</del></a> patch. It's not fixed. </p> <p>Cliff</p> <p>Please add the results of the debug logs to <a href="https://jira.whamcloud.com/browse/LU-9504" title="LBUG ptlrpc_handle_rs()) ASSERTION( lock != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-9504"><del>LU-9504</del></a> - this patch is unlanded to master so does not need a new ticket</p> <p>Peter</p> Duplicate LU-9504 Development Rank 1|hzzdnb: Rank (Obsolete) 9223372036854775807 Severity