[LU-3917] lod_initialize_objects()) ASSERTION( cfs_bitmap_check(md->lod_ost_descs.ltd_tgt_bitmap, idx) ) failed

• Type: Bug
• Resolution: Duplicate
• Priority: Blocker
• Affects Version/s: Lustre 2.4.0, Lustre 2.5.0
• Environment: lustre-2.4.0-15chaos_2.6.32_358.14.1.2chaos.ch5.1.1.ch5.1.1.x86_64
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Ned Bass (Inactive)

Description

MDS crashes with summary assertion after upgrade from 2.1 to 2.4. Added -o abort_recov mount option but MDS still crashes each time it is started.

PID: 4684   TASK: ffff880628b33540  CPU: 12  COMMAND: "tgt_recov"
 #0 [ffff880628b35a48] machine_kexec at ffffffff81035fcb
 #1 [ffff880628b35aa8] crash_kexec at ffffffff810c10b2
 #2 [ffff880628b35b78] panic at ffffffff81510333
 #3 [ffff880628b35bf8] lbug_with_loc at ffffffffa0507f4b [libcfs]
 #4 [ffff880628b35c18] lod_initialize_objects at ffffffffa1141d3b [lod]
 #5 [ffff880628b35ca8] lod_parse_striping at ffffffffa11421e1 [lod]
 #6 [ffff880628b35cd8] lod_load_striping at ffffffffa1143c44 [lod]
 #7 [ffff880628b35d18] lod_declare_object_destroy at ffffffffa114f6db [lod]
 #8 [ffff880628b35d48] __mdd_orphan_cleanup at ffffffffa0e190a9 [mdd]
 #9 [ffff880628b35de8] mdd_recovery_complete at ffffffffa0e2833d [mdd]
#10 [ffff880628b35e18] mdt_postrecov at ffffffffa1079cb5 [mdt]
#11 [ffff880628b35e38] mdt_obd_postrecov at ffffffffa107b178 [mdt]
#12 [ffff880628b35ea8] target_recovery_thread at ffffffffa09a6ca4 [ptlrpc]
#13 [ffff880628b35f48] kernel_thread at ffffffff8100c10a

ZFS: Loaded module v0.6.2-1.2, ZFS pool version 5000, ZFS filesystem version 5
Lustre: Lustre: Build Version: 2.4.0-15chaos-15chaos--PRISTINE-2.6.32-358.14.1.2chaos.ch5.1.1.x86_64
LDISKFS-fs (sdb): mounted filesystem with ordered data mode. quota=off. Opts: 
Lustre: lsc-MDT0000: Not available for connect from 192.168.117.178@o2ib10 (not set up)
Lustre: 4673:0:(mdt_handler.c:4947:mdt_process_config()) For interoperability, skip this mdd.quota_type. It is obsolete.
LustreError: 11-0: lsc-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.
LustreError: 4469:0:(mdt_handler.c:5930:mdt_iocontrol()) lsc-MDT0000: Aborting recovery for device
Lustre: lsc-MDT0000: Aborting recovery
LustreError: 4684:0:(lod_lov.c:706:lod_initialize_objects()) ASSERTION( cfs_bitmap_check(md->lod_ost_descs.ltd_tgt_bitmap, idx) ) failed: 
LustreError: 4684:0:(lod_lov.c:706:lod_initialize_objects()) LBUG
Pid: 4684, comm: tgt_recov

Call Trace:
 [<ffffffffa05078f5>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
 [<ffffffffa0507ef7>] lbug_with_loc+0x47/0xb0 [libcfs]
 [<ffffffffa1141d3b>] lod_initialize_objects+0x98b/0xc30 [lod]
 [<ffffffffa11421e1>] lod_parse_striping+0x201/0x300 [lod]
 [<ffffffffa1143c44>] lod_load_striping+0x2a4/0x4b0 [lod]
 [<ffffffffa114f6db>] lod_declare_object_destroy+0x16b/0x390 [lod]
 [<ffffffffa0e190a9>] __mdd_orphan_cleanup+0x7d9/0xca0 [mdd]
 [<ffffffffa0e2833d>] mdd_recovery_complete+0xed/0x170 [mdd]
 [<ffffffffa1079cb5>] mdt_postrecov+0x35/0xd0 [mdt]
 [<ffffffffa107b178>] mdt_obd_postrecov+0x78/0x90 [mdt]
 [<ffffffffa09964e0>] ? ldlm_reprocess_res+0x0/0x20 [ptlrpc]
 [<ffffffffa099189e>] ? ldlm_reprocess_all_ns+0x3e/0x110 [ptlrpc]
 [<ffffffffa09a6ca4>] target_recovery_thread+0xc64/0x1980 [ptlrpc]
 [<ffffffffa09a6040>] ? target_recovery_thread+0x0/0x1980 [ptlrpc]
 [<ffffffff8100c10a>] child_rip+0xa/0x20
 [<ffffffffa09a6040>] ? target_recovery_thread+0x0/0x1980 [ptlrpc]
 [<ffffffffa09a6040>] ? target_recovery_thread+0x0/0x1980 [ptlrpc]
 [<ffffffff8100c100>] ? child_rip+0x0/0x20

Kernel panic - not syncing: LBUG
Pid: 4684, comm: tgt_recov Tainted: P           ---------------    2.6.32-358.14.1.2chaos.ch5.1.1.x86_64 #1
Call Trace:
 [<ffffffff8151032c>] ? panic+0xa7/0x16f
 [<ffffffffa0507f4b>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
 [<ffffffffa1141d3b>] ? lod_initialize_objects+0x98b/0xc30 [lod]
 [<ffffffffa11421e1>] ? lod_parse_striping+0x201/0x300 [lod]
 [<ffffffffa1143c44>] ? lod_load_striping+0x2a4/0x4b0 [lod]
 [<ffffffffa114f6db>] ? lod_declare_object_destroy+0x16b/0x390 [lod]
 [<ffffffffa0e190a9>] ? __mdd_orphan_cleanup+0x7d9/0xca0 [mdd]
 [<ffffffffa0e2833d>] ? mdd_recovery_complete+0xed/0x170 [mdd]
 [<ffffffffa1079cb5>] ? mdt_postrecov+0x35/0xd0 [mdt]
 [<ffffffffa107b178>] ? mdt_obd_postrecov+0x78/0x90 [mdt]
 [<ffffffffa09964e0>] ? ldlm_reprocess_res+0x0/0x20 [ptlrpc]
 [<ffffffffa099189e>] ? ldlm_reprocess_all_ns+0x3e/0x110 [ptlrpc]
 [<ffffffffa09a6ca4>] ? target_recovery_thread+0xc64/0x1980 [ptlrpc]
 [<ffffffffa09a6040>] ? target_recovery_thread+0x0/0x1980 [ptlrpc]
 [<ffffffff8100c10a>] ? child_rip+0xa/0x20
 [<ffffffffa09a6040>] ? target_recovery_thread+0x0/0x1980 [ptlrpc]
 [<ffffffffa09a6040>] ? target_recovery_thread+0x0/0x1980 [ptlrpc]
 [<ffffffff8100c100>] ? child_rip+0x0/0x20
REWRITING MCP55 CFG REG
CFG = c1

Issue Links

• is duplicated by: LU-4005 Test failure on test suite recovery-double-scale, subtest test_pairwise_fail
• is related to: LU-3918 kernel NULL pointer dereference at lod_initialize_objects+0x327
• is related to: LU-3161 LASSERT() in lod_initialize_objects()