[LU-7384] lu_object_attr()) ASSERTION( ((o)->lo_header->loh_attr & LOHA_EXISTS) != 0

Related — Tue, 24 Nov 2015 14:44:24 +0000

Error happens during soak testing of build '20151027' (see 2.7.62, 5245b94a4aed1c4dcfa8b394acf916ea7accf138 ) + the following patches
DNE is enabled. MDTs had been formatted with ldiskfs, OSTs with zfs as backend FS. MDTs are configured in active-active HA failover configuration:

lola-8 (mdt0,1), lola-9 (mdt2,3)
lola-9 (mdt4,5), lola-11 (mdt6,7)

Due to problems described in ~~LU-7039~~ update_log, update_log_dirs have been deleted on all MDTs when mounted as ldiskfs devices.
Before starting soak the command

 lctl lfsck_start -A -M soaked-MDT0000 -t all

was exectuted at Oct, 27 08:00 (cluster time)
Almost immediately nodes lola-9,10 crashed with LBUG:

Oct 27 08:00:42 lola-9 kernel: LustreError: 7394:0:(lu_object.h:862:lu_object_attr()) ASSERTION( ((o)->lo_header->loh_attr &
 LOHA_EXISTS) != 0 ) failed:
Oct 27 08:00:42 lola-9 kernel: LustreError: 7394:0:(lu_object.h:862:lu_object_attr()) LBUG

-----------------------------------------------------------------------
Oct 27 08:00:42 lola-10 kernel: LustreError: 6078:0:(lu_object.h:862:lu_object_attr()) ASSERTION( ((o)->lo_header->loh_attr 
& LOHA_EXISTS) != 0 ) failed: 
Oct 27 08:00:42 lola-10 kernel: LustreError: 6078:0:(lu_object.h:862:lu_object_attr()) LBUG
Oct 27 08:00:42 lola-10 kernel: Pid: 6078, comm: lfsck_namespace
Oct 27 08:00:42 lola-10 kernel: 
Oct 27 08:00:42 lola-10 kernel: Call Trace:
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa0658875>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa0658e77>] lbug_with_loc+0x47/0xb0 [libcfs]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa107b06e>] lfsck_lock+0x3fe/0x430 [lfsck]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa12374d4>] ? lod_index_try+0x184/0x300 [lod]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa1099c86>] lfsck_namespace_insert_normal+0x306/0xa00 [lfsck]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa10a1a21>] lfsck_namespace_dsd_single+0x511/0xd40 [lfsck]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa10a5156>] lfsck_namespace_double_scan_dir+0x6d6/0xe40 [lfsck]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa10a5c14>] lfsck_namespace_double_scan_one+0x354/0x1330 [lfsck]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa07a58b1>] ? lu_object_find_at+0xb1/0xe0 [obdclass]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa07a49ed>] ? lu_object_put+0x25d/0x3b0 [obdclass]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa10afdcd>] lfsck_namespace_double_scan_one_trace_file+0x5bd/0x8d0 [lfsck]
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa10b045b>] lfsck_namespace_assistant_handler_p2+0x37b/0x1830 [lfsck]
Oct 27 08:00:42 lola-10 kernel: [<ffffffff81087540>] ? process_timeout+0x0/0x10
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa10924a3>] lfsck_assistant_engine+0x1633/0x2010 [lfsck]
Oct 27 08:00:42 lola-10 kernel: [<ffffffff81064c00>] ? default_wake_function+0x0/0x20
Oct 27 08:00:42 lola-10 kernel: [<ffffffffa1090e70>] ? lfsck_assistant_engine+0x0/0x2010 [lfsck]
Oct 27 08:00:42 lola-10 kernel: [<ffffffff8109e78e>] kthread+0x9e/0xc0
Oct 27 08:00:42 lola-10 kernel: [<ffffffff8100c28a>] child_rip+0xa/0x20
Oct 27 08:00:42 lola-10 kernel: [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
Oct 27 08:00:42 lola-10 kernel: [<ffffffff8100c280>] ? child_rip+0x0/0x20
Oct 27 08:00:42 lola-10 kernel: 
Oct 27 08:00:42 lola-10 kernel: Kernel panic - not syncing: LBUG

Crash dump files have been written have been written for the first event:

lola-9:/var/crash/127.0.0.1-2015-10-27-08:00:58
and can be provided on demand.

No corresponding log entries exist on any other node.

Whamcloud Community JIRA

[LU-7384] lu_object_attr()) ASSERTION( ((o)->lo_header->loh_attr & LOHA_EXISTS) != 0