[LU-2129] ASSERTION( last_rec->lrh_index == tail->lrt_index )

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.4.0
• Affects Version/s: Lustre 2.4.0
• Labels: NFBlocker
• Environment: https://build.whamcloud.com/job/lustre-master/951/ ; On opensfs with 1 mds, 4 oss, 24 clients
• Severity: 3
• Assignee: Mikhail Pershin
• Reporter: Minh Diep

Description

Lustre: DEBUG MARKER: ----============= acceptance-small: parallel-scale ============---- Tue Oct 9 09:38:01 PDT 2012
Lustre: DEBUG MARKER: excepting tests: parallel_grouplock
Lustre: DEBUG MARKER: Using TIMEOUT=20
LustreError: 13309:0:(osd_io.c:1062:osd_ldiskfs_write_record()) sdd1: error reading offset 12288 (block 3): rc = -28
LustreError: 13309:0:(llog_osd.c:175:llog_osd_write_blob()) lustre-MDT0000-osd: error writing log buffer: rc = -28
LustreError: 13309:0:(mgs_llog.c:738:record_lcfg()) failed -28
LustreError: 13309:0:(mgs_llog.c:1074:mgs_write_log_direct_all()) MGS: writing log lustre-params: rc = -28
LustreError: 13309:0:(osd_io.c:1062:osd_ldiskfs_write_record()) sdd1: error reading offset 12288 (block 3): rc = -28
LustreError: 13309:0:(llog_osd.c:175:llog_osd_write_blob()) lustre-MDT0000-osd: error writing log buffer: rc = -28
LustreError: 13309:0:(mgs_llog.c:738:record_lcfg()) failed -28
LustreError: 13309:0:(mgs_llog.c:1074:mgs_write_log_direct_all()) MGS: writing log lustre-OST0000: rc = -28
LustreError: 13309:0:(mgs_llog.c:2956:mgs_write_log_param()) err -28 on param 'sys.jobid_var=procname_uid'
LustreError: 13309:0:(mgs_handler.c:926:mgs_iocontrol()) MGS: setparam err: rc = -28
LustreError: 11580:0:(llog_osd.c:579:llog_osd_next_block()) ASSERTION( last_rec->lrh_index == tail->lrt_index ) failed:
LustreError: 11580:0:(llog_osd.c:579:llog_osd_next_block()) LBUG
Pid: 11580, comm: ll_mgs_0003

Call Trace:
[<ffffffffa0991905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
[<ffffffffa0991f17>] lbug_with_loc+0x47/0xb0 [libcfs]
[<ffffffffa0cd6435>] llog_osd_next_block+0xa55/0xa60 [obdclass]
[<ffffffffa151d436>] ? lustre_pack_reply_flags+0xb6/0x210 [ptlrpc]
[<ffffffffa153a80c>] llog_origin_handle_next_block+0x50c/0x780 [ptlrpc]
[<ffffffffa1747f93>] mgs_handle+0xb13/0x11e0 [mgs]
[<ffffffffa09a2621>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa152d8dc>] ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc]
[<ffffffff8104e309>] ? __wake_up_common+0x59/0x90
[<ffffffff810533f3>] ? __wake_up+0x53/0x70
[<ffffffffa152eecc>] ptlrpc_main+0xc0c/0x19f0 [ptlrpc]
[<ffffffffa152e2c0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
[<ffffffff8100c14a>] child_rip+0xa/0x20
[<ffffffffa152e2c0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
[<ffffffffa152e2c0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
[<ffffffff8100c140>] ? child_rip+0x0/0x20

Kernel panic - not syncing: LBUG
Pid: 11580, comm: ll_mgs_0003 Not tainted 2.6.32-279.5.1.el6_lustre.x86_64 #1
Call Trace:
[<ffffffff814fd58a>] ? panic+0xa0/0x168
[<ffffffffa0991f6b>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
[<ffffffffa0cd6435>] ? llog_osd_next_block+0xa55/0xa60 [obdclass]
[<ffffffffa151d436>] ? lustre_pack_reply_flags+0xb6/0x210 [ptlrpc]
[<ffffffffa153a80c>] ? llog_origin_handle_next_block+0x50c/0x780 [ptlrpc]
[<ffffffffa1747f93>] ? mgs_handle+0xb13/0x11e0 [mgs]
[<ffffffffa09a2621>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa152d8dc>] ? ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc]
[<ffffffff8104e309>] ? __wake_up_common+0x59/0x90
[<ffffffff810533f3>] ? __wake_up+0x53/0x70
[<ffffffffa152eecc>] ? ptlrpc_main+0xc0c/0x19f0 [ptlrpc]
[<ffffffffa152e2c0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
[<ffffffff8100c14a>] ? child_rip+0xa/0x20
[<ffffffffa152e2c0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
[<ffffffffa152e2c0>] ? ptlrpc_main+0x0/0x19f0 [ptlrpc]
[<ffffffff8100c140>] ? child_rip+0x0/0x20