[LU-6963] racer test_1: lu_object_attr() ASSERTION(lo_header->loh_attr & LOHA_EXISTS) failed in rename

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Major
• Reporter: Maloo
• Assignee: WC Triage
• Resolution: Unresolved
• Labels: zfs

Environment

client and server: lustre-master build # 3118 RHEL7.1 ZFS

• Severity: 3

Description

This issue was created by maloo for sarah_lw <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/fac93ec6-3b4b-11e5-acaf-5254006e85c2.

The sub-test test_1 failed with the following error:

test failed to respond and timed out

06:37:57:[24383.593537] LustreError: 6406:0:(mdd_object.c:70:mdd_la_get()) lustre-MDD0000: object [0x200000402:0x1ab4:0x0] not found: rc = -2
06:37:57:[24383.599610] LustreError: 6406:0:(mdd_object.c:70:mdd_la_get()) Skipped 9 previous similar messages
06:38:09:[24387.052197] LustreError: 6414:0:(osd_handler.c:222:osd_trans_start()) lustre-MDT0000: can't assign tx: rc = -17
06:38:09:[24399.879030] LustreError: 6395:0:(lu_object.h:861:lu_object_attr()) ASSERTION( ((o)->lo_header->loh_attr & LOHA_EXISTS) != 0 ) failed: 
06:38:09:[24399.884611] LustreError: 6395:0:(lu_object.h:861:lu_object_attr()) LBUG
06:38:09:[24399.888047] Pid: 6395, comm: mdt00_010
06:38:09:[24399.890770] 
06:38:09:[24399.890770] Call Trace:
06:38:09:[24399.895375]  [<ffffffffa08207d3>] libcfs_debug_dumpstack+0x53/0x80 [libcfs]
06:38:09:[24399.898101]  [<ffffffffa0820d75>] lbug_with_loc+0x45/0xc0 [libcfs]
06:38:09:[24399.900957]  [<ffffffffa10ba9d9>] dt_declare_record_write.part.14+0x0/0x36 [mdd]
06:38:09:[24399.903598]  [<ffffffffa109db1e>] mdd_is_parent.isra.26+0x64e/0x6c0 [mdd]
06:38:09:[24399.906118]  [<ffffffffa0ddfa4c>] ? osd_attr_get+0xfc/0x1d0 [osd_zfs]
06:38:09:[24399.908814]  [<ffffffffa109dd5d>] mdd_is_subdir+0x1cd/0x240 [mdd]
06:38:09:[24399.911410]  [<ffffffffa0f6c6ba>] mdt_is_subdir.isra.32+0x16a/0x330 [mdt]
06:38:09:[24399.913881]  [<ffffffffa0f6f5dc>] mdt_reint_rename_internal.isra.36+0xf6c/0x1c00 [mdt]
06:38:09:[24399.916642]  [<ffffffffa0b9d371>] ? ldlm_cli_enqueue_local+0x271/0x940 [ptlrpc]
06:38:09:[24399.919228]  [<ffffffffa0f7161b>] mdt_reint_rename_or_migrate.isra.39+0x19b/0x850 [mdt]
06:38:09:[24399.921905]  [<ffffffffa0ba1a10>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
06:38:09:[24399.924530]  [<ffffffffa0b9bfb0>] ? ldlm_completion_ast+0x0/0x910 [ptlrpc]
06:38:09:[24399.927136]  [<ffffffffa0f71d03>] mdt_reint_rename+0x13/0x20 [mdt]
06:38:09:[24399.929513]  [<ffffffffa0f75b00>] mdt_reint_rec+0x80/0x210 [mdt]
06:38:09:[24399.931976]  [<ffffffffa0f59619>] mdt_reint_internal+0x5d9/0xab0 [mdt]
06:38:09:[24399.934268]  [<ffffffffa0f62a27>] mdt_reint+0x67/0x140 [mdt]
06:38:09:[24399.936565]  [<ffffffffa0c34a0b>] tgt_request_handle+0x88b/0x1100 [ptlrpc]
06:38:09:[24399.938888]  [<ffffffffa0bdd7cb>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]
06:38:09:[24399.941181]  [<ffffffffa0bda8f8>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
06:38:09:[24399.943336]  [<ffffffff810a9662>] ? default_wake_function+0x12/0x20
06:38:09:[24399.945504]  [<ffffffff810a0898>] ? __wake_up_common+0x58/0x90
06:38:09:[24399.947533]  [<ffffffffa0be0f70>] ptlrpc_main+0xb70/0x1ed0 [ptlrpc]
06:38:09:[24399.949670]  [<ffffffff810ad8b6>] ? __dequeue_entity+0x26/0x40
06:38:09:[24399.951583]  [<ffffffff810125f6>] ? __switch_to+0x136/0x4a0
06:38:09:[24399.953622]  [<ffffffff81609fc5>] ? __schedule+0x2c5/0x7b0
06:38:09:[24399.956136]  [<ffffffffa0be0400>] ? ptlrpc_main+0x0/0x1ed0 [ptlrpc]
06:38:09:[24399.958785]  [<ffffffff8109739f>] kthread+0xcf/0xe0
06:38:09:[24399.960951]  [<ffffffff810972d0>] ? kthread+0x0/0xe0
06:38:09:[24399.963406]  [<ffffffff81615018>] ret_from_fork+0x58/0x90
06:38:09:[24399.965393]  [<ffffffff810972d0>] ? kthread+0x0/0xe0
06:38:09:[24399.967326] 
06:38:09:[24399.969060] Kernel panic - not syncing: LBUG
06:38:09:[24399.970027] CPU: 0 PID: 6395 Comm: mdt00_010 Tainted: PF          O--------------   3.10.0-229.7.2.el7_lustre.gfd6f11c.x86_64 #1
06:38:09:[24399.970027] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
06:38:09:[24399.970027]  ffffffffa083dfaf 000000008255d518 ffff8800581e78d0 ffffffff816051aa
06:38:09:[24399.970027]  ffff8800581e7950 ffffffff815fea1e ffffffff00000008 ffff8800581e7960
06:38:09:[24399.970027]  ffff8800581e7900 000000008255d518 ffffffffa10bcd6d 0000000000000246
06:38:09:[24399.970027] Call Trace:
06:38:09:[24399.970027]  [<ffffffff816051aa>] dump_stack+0x19/0x1b
06:38:09:[24399.970027]  [<ffffffff815fea1e>] panic+0xd8/0x1e7
06:38:09:[24399.984529]  [<ffffffffa0820ddb>] lbug_with_loc+0xab/0xc0 [libcfs]
06:38:09:[24399.984529]  [<ffffffffa10ba9d9>] lu_object_attr.isra.11.part.12+0x36/0x36 [mdd]
06:38:09:[24399.984529]  [<ffffffffa109db1e>] mdd_is_parent.isra.26+0x64e/0x6c0 [mdd]
06:38:09:[24399.984529]  [<ffffffffa0ddfa4c>] ? osd_attr_get+0xfc/0x1d0 [osd_zfs]
06:38:09:[24399.984529]  [<ffffffffa109dd5d>] mdd_is_subdir+0x1cd/0x240 [mdd]
06:38:09:[24399.984529]  [<ffffffffa0f6c6ba>] mdt_is_subdir.isra.32+0x16a/0x330 [mdt]
06:38:09:[24399.984529]  [<ffffffffa0f6f5dc>] mdt_reint_rename_internal.isra.36+0xf6c/0x1c00 [mdt]
06:38:09:[24399.996138]  [<ffffffffa0b9d371>] ? ldlm_cli_enqueue_local+0x271/0x940 [ptlrpc]
06:38:09:[24399.996138]  [<ffffffffa0f7161b>] mdt_reint_rename_or_migrate.isra.39+0x19b/0x850 [mdt]
06:38:09:[24399.996138]  [<ffffffffa0ba1a10>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
06:38:09:[24399.996138]  [<ffffffffa0b9bfb0>] ? ldlm_expired_completion_wait+0x370/0x370 [ptlrpc]
06:38:09:[24399.996138]  [<ffffffffa0f71d03>] mdt_reint_rename+0x13/0x20 [mdt]
06:38:09:[24399.996138]  [<ffffffffa0f75b00>] mdt_reint_rec+0x80/0x210 [mdt]
06:38:09:[24399.996138]  [<ffffffffa0f59619>] mdt_reint_internal+0x5d9/0xab0 [mdt]
06:38:09:[24399.996138]  [<ffffffffa0f62a27>] mdt_reint+0x67/0x140 [mdt]
06:38:09:[24399.996138]  [<ffffffffa0c34a0b>] tgt_request_handle+0x88b/0x1100 [ptlrpc]
06:38:09:[24399.996138]  [<ffffffffa0bdd7cb>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]
06:38:09:[24399.996138]  [<ffffffffa0bda8f8>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
06:38:09:[24399.996138]  [<ffffffff810a9662>] ? default_wake_function+0x12/0x20
06:38:09:[24399.996138]  [<ffffffff810a0898>] ? __wake_up_common+0x58/0x90
06:38:09:[24399.996138]  [<ffffffffa0be0f70>] ptlrpc_main+0xb70/0x1ed0 [ptlrpc]
06:38:09:[24399.996138]  [<ffffffff810ad8b6>] ? __dequeue_entity+0x26/0x40
06:38:09:[24399.996138]  [<ffffffff810125f6>] ? __switch_to+0x136/0x4a0
06:38:09:[24399.996138]  [<ffffffff81609fc5>] ? __schedule+0x2c5/0x7b0
06:38:09:[24399.996138]  [<ffffffffa0be0400>] ? ptlrpc_register_service+0xfc0/0xfc0 [ptlrpc]
06:38:09:[24399.996138]  [<ffffffff8109739f>] kthread+0xcf/0xe0
06:38:09:[24399.996138]  [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140
06:38:09:[24399.996138]  [<ffffffff81615018>] ret_from_fork+0x58/0x90
06:38:09:[24399.996138]  [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140
06:38:09:[24399.996138] drm_kms_helper: panic occurred, switching back to text console
06:38:09:[24399.996138] ------------[ cut here ]------------

Issue Links

• is related to: LU-7145 mdd_object_type() uses need audit
• is related to: LU-14457 racer test_1: crash with directory mi...
• is related to: LU-7617 replay-single test_70b hangs ASSERTIO...

Comments

[Mikhail Pershin]

another occurrence in replay-single test_70b, ldiskfs now:

https://testing.hpdd.intel.com/test_sets/878cad84-3b2e-11e5-95fa-5254006e85c2

20:51:01:LustreError: 15616:0:(lu_object.h:861:lu_object_attr()) ASSERTION( ((o)->lo_header->loh_attr & LOHA_EXISTS) != 0 ) failed: 
20:51:01:LustreError: 15616:0:(lu_object.h:861:lu_object_attr()) LBUG
20:51:01:Pid: 15616, comm: mdt00_001
20:51:01:
20:51:01:Call Trace:
20:51:01: [<ffffffffa0490875>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
20:51:01: [<ffffffffa0490e77>] lbug_with_loc+0x47/0xb0 [libcfs]
20:51:01: [<ffffffffa1042260>] mdd_rename+0x1ba0/0x1e10 [mdd]
20:51:01: [<ffffffffa103fb80>] ? __mdd_lookup+0x250/0x4e0 [mdd]
20:51:01: [<ffffffffa0efb105>] mdt_reint_rename_internal+0x1305/0x1a50 [mdt]
20:51:01: [<ffffffffa07c6246>] ? ldlm_lock_enqueue+0x2c6/0x8e0 [ptlrpc]
20:51:01: [<ffffffffa0efba4d>] mdt_reint_rename_or_migrate+0x1fd/0x7e0 [mdt]
20:51:01: [<ffffffffa07e5430>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
20:51:01: [<ffffffffa07e6da0>] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc]
20:51:01: [<ffffffffa083c3b2>] ? __req_capsule_get+0x162/0x6e0 [ptlrpc]
20:51:01: [<ffffffff81294a3a>] ? strlcpy+0x4a/0x60
20:51:01: [<ffffffffa0efc063>] mdt_reint_rename+0x13/0x20 [mdt]
20:51:01: [<ffffffffa0ef486d>] mdt_reint_rec+0x5d/0x200 [mdt]
20:51:01: [<ffffffffa0ee078b>] mdt_reint_internal+0x62b/0xb80 [mdt]
20:51:01: [<ffffffffa0ee117b>] mdt_reint+0x6b/0x120 [mdt]
20:51:01: [<ffffffffa087dfb2>] tgt_request_handle+0xa42/0x1230 [ptlrpc]
20:51:01: [<ffffffffa0826191>] ptlrpc_main+0xe41/0x1920 [ptlrpc]
20:51:01: [<ffffffffa0825350>] ? ptlrpc_main+0x0/0x1920 [ptlrpc]
20:51:01: [<ffffffff8109e78e>] kthread+0x9e/0xc0
20:51:01: [<ffffffff8100c28a>] child_rip+0xa/0x20
20:51:01: [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
20:51:01: [<ffffffff8100c280>] ? child_rip+0x0/0x20
20:51:01:
20:51:01:Kernel panic - not syncing: LBUG