https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10160 Lustre <p>After OSS failover and recovery:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Oct 25 08:41:02 soak-2 kernel: Lustre: soaked-OST0012: Recovery over after 0:35, of 33 clients 33 recovered and 0 were evicted. Oct 25 08:41:03 soak-2 kernel: LustreError: 26237:0:(ofd_io.c:616:ofd_preprw_write()) soaked-OST0012: BRW to missing obj 0x0:1943200 Oct 25 08:41:03 soak-2 kernel: LustreError: 26369:0:(lustre_dlm.h:1371:ldlm_res_lvbo_update()) delayed lvb init failed (rc -2) Oct 25 08:41:03 soak-2 kernel: format at lustre_dlm.h:1097:ldlm_lvbo_fill doesn't end in newline Oct 25 08:41:03 soak-2 kernel: LustreError: 26237:0:(ofd_io.c:616:ofd_preprw_write()) Skipped 3 previous similar messages Oct 25 08:41:03 soak-2 kernel: format at lustre_dlm.h:1097:ldlm_lvbo_fill doesn't end in newline Oct 25 08:41:03 soak-2 kernel: LustreError: 12692:0:(ofd_io.c:616:ofd_preprw_write()) soaked-OST0012: BRW to missing obj 0x780000400:835459 Oct 25 08:41:03 soak-2 kernel: LustreError: 12692:0:(ofd_io.c:616:ofd_preprw_write()) Skipped 1 previous similar message Oct 25 08:41:03 soak-2 kernel: Lustre: soaked-OST0006: Connection restored to e5f19f2d-79a3-0744-c0c5-e9d37cbaf536 (at 192.168.1.118@o2ib) Oct 25 08:41:03 soak-2 kernel: Lustre: soaked-OST0000: Connection restored to e5f19f2d-79a3-0744-c0c5-e9d37cbaf536 (at 192.168.1.118@o2ib) Oct 25 08:41:03 soak-2 kernel: Lustre: Skipped 43 previous similar messages Oct 25 08:41:03 soak-2 kernel: Lustre: Skipped 43 previous similar messages Oct 25 08:41:03 soak-2 kernel: LustreError: 26369:0:(lustre_dlm.h:1097:ldlm_lvbo_fill()) lock ffff8800a4621c00: delayed lvb init failed (rc -2) Oct 25 08:41:03 soak-2 kernel: LustreError: 26469:0:(lustre_dlm.h:1097:ldlm_lvbo_fill()) lock ffff8800afe75200: delayed lvb init failed (rc -2) Oct 25 08:41:03 soak-2 kernel: LustreError: 26469:0:(ldlm_lock.c:2259:__ldlm_reprocess_all()) ASSERTION( list_empty(&amp;rpc_list) ) failed: Oct 25 08:41:04 soak-2 kernel: LustreError: 26469:0:(ldlm_lock.c:2259:__ldlm_reprocess_all()) LBUG Oct 25 08:41:04 soak-2 kernel: Pid: 26469, comm: ldlm_cn00_003 Oct 25 08:41:04 soak-2 kernel: #012Call Trace: Oct 25 08:41:04 soak-2 kernel: [&lt;ffffffffc0d3e7ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Oct 25 08:41:04 soak-2 kernel: [&lt;ffffffffc0d3e83c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Oct 25 08:41:04 soak-2 kernel: [&lt;ffffffffc112a88b&gt;] __ldlm_reprocess_all+0x32b/0x3a0 [ptlrpc] Oct 25 08:41:04 soak-2 kernel: [&lt;ffffffffc112a910&gt;] ldlm_reprocess_all+0x10/0x20 [ptlrpc] Oct 25 08:41:04 soak-2 kernel: [&lt;ffffffffc114dc8d&gt;] ldlm_request_cancel+0x46d/0x720 [ptlrpc] Oct 25 08:41:04 soak-2 kernel: [&lt;ffffffffc115417a&gt;] ldlm_handle_cancel+0xba/0x250 [ptlrpc] Oct 25 08:41:04 soak-2 kernel: [&lt;ffffffffc1154451&gt;] ldlm_cancel_handler+0x141/0x490 [ptlrpc] Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffffc118537e&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffffc1182248&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffff810c4822&gt;] ? default_wake_function+0x12/0x20 Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffff810ba588&gt;] ? __wake_up_common+0x58/0x90 Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffffc1188b22&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffff81029557&gt;] ? __switch_to+0xd7/0x510 Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffff816a8f01&gt;] ? __schedule+0x311/0x8b0 Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffffc1188090&gt;] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 Oct 25 08:41:05 soak-2 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 </pre> </div></div> <p>Lustre-log dump attached</p> Soak cluster version=2.10.54_13_g84f690e<br/> lustre-master-ib build 6 with DNE configured LU-10160

LBUG ldlm_lock.c:2259:__ldlm_reprocess_all()) ASSERTION( list_empty(&rpc_list) ) failed:

Bug Major Open Unresolved Oleg Drokin Cliff White soak Wed, 25 Oct 2017 15:08:47 +0000 Wed, 27 Nov 2019 16:02:22 +0000 Lustre 2.11.0 0 5 <p>Niu - Would you please review this error and comment on possible causes?</p> <p>Thanks</p> <p>We do have multiple earlier hits of the 'BRW to missing obj' <br/> Example: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">ct 12 08:29:22 soak-3 kernel: Lustre: soaked-OST000d: Will be in recovery <span class="code-keyword">for</span> at least 2:30, or until 34 clients reconnect Oct 12 08:29:29 soak-3 kernel: Lustre: soaked-OST000d: Connection restored to 19d86900-89d1-0961-b2cc-97d38ab6ed62 (at 192.168.1.137@o2ib) Oct 12 08:29:29 soak-3 kernel: Lustre: Skipped 68 previous similar messages Oct 12 08:29:40 soak-3 kernel: Lustre: soaked-OST0013: Recovery over after 0:32, of 34 clients 34 recovered and 0 were evicted. Oct 12 08:29:41 soak-3 kernel: LustreError: 9714:0:(ofd_io.c:616:ofd_preprw_write()) soaked-OST0007: BRW to missing obj 0x0:51358755 Oct 12 08:29:52 soak-3 kernel: LustreError: 43590:0:(osd_index.c:223:__osd_xattr_load_by_oid()) soaked-OST000d: can't get bonus, rc = -17 Oct 12 08:29:53 soak-3 kernel: Lustre: 9643:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> slow reply: [sent 1507796987/real 1507796987] req@ffff880816ff1200 x1581039691499472/t0(0) o38-&gt;soaked-MDT0000-lwp-OST000d@192.168.1.109@o2ib:12/10 lens 520/544 e 0 to 1 dl 1507796993 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 ........ Oct 13 13:46:50 soak-4 kernel: Lustre: soaked-OST000f: Will be in recovery <span class="code-keyword">for</span> at least 2:30, or until 34 clients reconnect Oct 13 13:46:57 soak-4 kernel: Lustre: soaked-OST0009: Recovery over after 1:02, of 34 clients 34 recovered and 0 were evicted. Oct 13 13:46:57 soak-4 kernel: LustreError: 9585:0:(ofd_io.c:616:ofd_preprw_write()) soaked-OST0009: BRW to missing obj 0x0:67095807 Oct 13 13:46:57 soak-4 kernel: LustreError: 9585:0:(ofd_io.c:616:ofd_preprw_write()) Skipped 10 previous similar messages </pre> </div></div> <p>Hm, another unrelated stuff from the logs, seems the obd to target name is not working especially well:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00080000:1.0:1508920863.284572:0:26256:0:(import.c:104:ptlrpc_import_enter_resend()) ffff880179734000 0&lt;E4&gt;&lt;CF&gt;&lt;EE&gt;^G&lt;88&gt;&lt;FF&gt;&lt;FF&gt;0&lt;E4&gt;&lt;CF&gt;&lt;EE&gt;^G&lt;88&gt;&lt;FF&gt;&lt;FF&gt;: changing import state from NEW to RECOVER 00000100:00080000:1.0:1508920863.284578:0:26256:0:(import.c:1541:ptlrpc_import_recovery_state_machine()) ffff880179734000 0&lt;E4&gt;&lt;CF&gt;&lt;EE&gt;^G&lt;88&gt;&lt;FF&gt;&lt;FF&gt;0&lt;E4&gt;&lt;CF&gt;&lt;EE&gt;^G&lt;88&gt;&lt;FF&gt;&lt;FF&gt;: changing import state from RECOVER to FULL </pre> </div></div> <p>Crashed again with 2.10.55, vmcore-dmesg attached, crash dump available on soak. </p> <p>Crashed again after full weekend, multiple successful failovers. Crash dump available on spirit. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[37601.153508] Lustre: soaked-OST000c: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [37601.282466] Lustre: soaked-OST000c: in recovery but waiting <span class="code-keyword">for</span> the first client to connect [37601.515428] Lustre: soaked-OST000c: Will be in recovery <span class="code-keyword">for</span> at least 2:30, or until 33 clients reconnect [37610.255005] Lustre: soaked-OST0012: Recovery over after 0:52, of 33 clients 33 recovered and 0 were evicted. [37610.255159] LustreError: 5409:0:(ofd_io.c:671:ofd_preprw_write()) soaked-OST0012: BRW to missing obj 0x740000402:491210 [37610.256286] Lustre: soaked-OST0012: deleting orphan objects from 0x0:830355 to 0x0:831537 [37610.258346] LustreError: 5387:0:(lustre_dlm.h:1376:ldlm_lvbo_update()) delayed lvb init failed (rc -2) [37610.509648] format at lustre_dlm.h:1103:ldlm_lvbo_fill doesn't end in newline [37610.509688] LustreError: 157856:0:(lustre_dlm.h:1103:ldlm_lvbo_fill()) lock ffff880820f860c0: delayed lvb init failed (rc -2) [37610.509688] LustreError: 157856:0:(ldlm_lock.c:2244:__ldlm_reprocess_all()) ASSERTION( list_empty(&amp;rpc_list) ) failed: [37610.509691] LustreError: 157856:0:(ldlm_lock.c:2244:__ldlm_reprocess_all()) LBUG [37610.509693] Pid: 157856, comm: ldlm_cn00_011 [37610.509693] Call Trace: [37610.509715] [&lt;ffffffffc0a477ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [37610.509726] [&lt;ffffffffc0a4783c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [37610.509795] [&lt;ffffffffc0ddd60b&gt;] __ldlm_reprocess_all+0x30b/0x380 [ptlrpc] [37610.509846] [&lt;ffffffffc0ddd690&gt;] ldlm_reprocess_all+0x10/0x20 [ptlrpc] [37610.509904] [&lt;ffffffffc0e002ad&gt;] ldlm_request_cancel+0x41d/0x710 [ptlrpc] [37610.509962] [&lt;ffffffffc0e0534a&gt;] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [37610.510018] [&lt;ffffffffc0e05621&gt;] ldlm_cancel_handler+0x141/0x490 [ptlrpc] [37610.510104] [&lt;ffffffffc0e3658e&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [37610.510168] [&lt;ffffffffc0e33448&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [37610.510174] [&lt;ffffffff810c6452&gt;] ? default_wake_function+0x12/0x20 [37610.510179] [&lt;ffffffff810bc0f8&gt;] ? __wake_up_common+0x58/0x90 [37610.510242] [&lt;ffffffffc0e39d42&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [37610.510305] [&lt;ffffffffc0e392b0&gt;] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [37610.510310] [&lt;ffffffff810b252f&gt;] kthread+0xcf/0xe0 [37610.510315] [&lt;ffffffff810c0d34&gt;] ? finish_task_switch+0x54/0x160 [37610.510318] [&lt;ffffffff810b2460&gt;] ? kthread+0x0/0xe0 [37610.510323] [&lt;ffffffff816b8798&gt;] ret_from_fork+0x58/0x90 [37610.510326] [&lt;ffffffff810b2460&gt;] ? kthread+0x0/0xe0 [37610.510327] [37610.510328] Kernel panic - not syncing: LBUG </pre> </div></div> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36885" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36885</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10160" title="LBUG ldlm_lock.c:2259:__ldlm_reprocess_all()) ASSERTION( list_empty(&amp;rpc_list) ) failed:" class="issue-link" data-issue-key="LU-10160">LU-10160</a> ldlm: fix message of ldlm_lvbo_fill<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d7eef551256205643ee70fe8cd102498470cae1d</p> Related Development Rank 1|hzzmfr: Rank (Obsolete) 9223372036854775807 Severity