[LU-10160] LBUG ldlm_lock.c:2259:__ldlm_reprocess_all()) ASSERTION( list_empty(&rpc_list) ) failed:

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.11.0
• Labels: soak
• Environment: Soak cluster version=2.10.54_13_g84f690e; lustre-master-ib build 6 with DNE configured
• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Cliff White (Inactive)

Description

After OSS failover and recovery:

Oct 25 08:41:02 soak-2 kernel: Lustre: soaked-OST0012: Recovery over after 0:35, of 33 clients 33 recovered and 0 were evicted.
Oct 25 08:41:03 soak-2 kernel: LustreError: 26237:0:(ofd_io.c:616:ofd_preprw_write()) soaked-OST0012: BRW to missing obj 0x0:1943200
Oct 25 08:41:03 soak-2 kernel: LustreError: 26369:0:(lustre_dlm.h:1371:ldlm_res_lvbo_update()) delayed lvb init failed (rc -2)
Oct 25 08:41:03 soak-2 kernel: format at lustre_dlm.h:1097:ldlm_lvbo_fill doesn't end in newline
Oct 25 08:41:03 soak-2 kernel: LustreError: 26237:0:(ofd_io.c:616:ofd_preprw_write()) Skipped 3 previous similar messages
Oct 25 08:41:03 soak-2 kernel: format at lustre_dlm.h:1097:ldlm_lvbo_fill doesn't end in newline
Oct 25 08:41:03 soak-2 kernel: LustreError: 12692:0:(ofd_io.c:616:ofd_preprw_write()) soaked-OST0012: BRW to missing obj 0x780000400:835459
Oct 25 08:41:03 soak-2 kernel: LustreError: 12692:0:(ofd_io.c:616:ofd_preprw_write()) Skipped 1 previous similar message
Oct 25 08:41:03 soak-2 kernel: Lustre: soaked-OST0006: Connection restored to e5f19f2d-79a3-0744-c0c5-e9d37cbaf536 (at 192.168.1.118@o2ib)
Oct 25 08:41:03 soak-2 kernel: Lustre: soaked-OST0000: Connection restored to e5f19f2d-79a3-0744-c0c5-e9d37cbaf536 (at 192.168.1.118@o2ib)
Oct 25 08:41:03 soak-2 kernel: Lustre: Skipped 43 previous similar messages
Oct 25 08:41:03 soak-2 kernel: Lustre: Skipped 43 previous similar messages
Oct 25 08:41:03 soak-2 kernel: LustreError: 26369:0:(lustre_dlm.h:1097:ldlm_lvbo_fill()) lock ffff8800a4621c00: delayed lvb init failed (rc -2)
Oct 25 08:41:03 soak-2 kernel: LustreError: 26469:0:(lustre_dlm.h:1097:ldlm_lvbo_fill()) lock ffff8800afe75200: delayed lvb init failed (rc -2)
Oct 25 08:41:03 soak-2 kernel: LustreError: 26469:0:(ldlm_lock.c:2259:__ldlm_reprocess_all()) ASSERTION( list_empty(&rpc_list) ) failed:
Oct 25 08:41:04 soak-2 kernel: LustreError: 26469:0:(ldlm_lock.c:2259:__ldlm_reprocess_all()) LBUG
Oct 25 08:41:04 soak-2 kernel: Pid: 26469, comm: ldlm_cn00_003
Oct 25 08:41:04 soak-2 kernel: #012Call Trace:
Oct 25 08:41:04 soak-2 kernel: [<ffffffffc0d3e7ae>] libcfs_call_trace+0x4e/0x60 [libcfs]
Oct 25 08:41:04 soak-2 kernel: [<ffffffffc0d3e83c>] lbug_with_loc+0x4c/0xb0 [libcfs]
Oct 25 08:41:04 soak-2 kernel: [<ffffffffc112a88b>] __ldlm_reprocess_all+0x32b/0x3a0 [ptlrpc]
Oct 25 08:41:04 soak-2 kernel: [<ffffffffc112a910>] ldlm_reprocess_all+0x10/0x20 [ptlrpc]
Oct 25 08:41:04 soak-2 kernel: [<ffffffffc114dc8d>] ldlm_request_cancel+0x46d/0x720 [ptlrpc]
Oct 25 08:41:04 soak-2 kernel: [<ffffffffc115417a>] ldlm_handle_cancel+0xba/0x250 [ptlrpc]
Oct 25 08:41:04 soak-2 kernel: [<ffffffffc1154451>] ldlm_cancel_handler+0x141/0x490 [ptlrpc]
Oct 25 08:41:05 soak-2 kernel: [<ffffffffc118537e>] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc]
Oct 25 08:41:05 soak-2 kernel: [<ffffffffc1182248>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Oct 25 08:41:05 soak-2 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Oct 25 08:41:05 soak-2 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Oct 25 08:41:05 soak-2 kernel: [<ffffffffc1188b22>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Oct 25 08:41:05 soak-2 kernel: [<ffffffff81029557>] ? __switch_to+0xd7/0x510
Oct 25 08:41:05 soak-2 kernel: [<ffffffff816a8f01>] ? __schedule+0x311/0x8b0
Oct 25 08:41:05 soak-2 kernel: [<ffffffffc1188090>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Oct 25 08:41:05 soak-2 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Oct 25 08:41:05 soak-2 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Oct 25 08:41:05 soak-2 kernel: [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
Oct 25 08:41:05 soak-2 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0

Lustre-log dump attached