[LU-9452] sanityn test_29 Failed: (ldlm_lock.c:1736:ldlm_lock_enqueue()) LBUG

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.11.0
• Affects Version/s: Lustre 2.10.0
• Labels: medium
• Severity: 3
• Assignee: WC Triage
• Reporter: Saurabh Tandan (Inactive)

Description

Sanityn test_29 passing on autotest when removed from Always_Except list as part of LU-7108 but was failing 100% of times on Oleg's testing environment.
https://review.whamcloud.com/#/c/16383/

 [ 7123.813636] Lustre: DEBUG MARKER: == sanityn test 29: lock put race between glimpse and enqueue ======================================== 03:06:11 (1492758371)
 [ 7123.852810] LustreError: 23298:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 30f sleeping for 2000ms
 [ 7123.854109] BUG: scheduling while atomic: ll_ost00_007/23298/0x00000002
 [ 7123.854877] INFO: lockdep is turned off.
 [ 7123.855549] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 loop mbcache jbd2 rpcsec_gss_krb5 syscopyarea sysfillrect sysimgblt ttm drm_kms_helper ata_generic pata_acpi drm i2c_piix4 ata_piix serio_raw pcspkr i2c_core virtio_balloon virtio_console libata virtio_blk floppy nfsd ip_tables [last unloaded: libcfs]
 [ 7123.861765] CPU: 3 PID: 23298 Comm: ll_ost00_007 Tainted: G           OE  ------------   3.10.0-debug #1
 [ 7123.863008] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
 [ 7123.881153]  ffff88002a984600 000000005186e0fc ffff88002c0538f8 ffffffff816fd400
 [ 7123.882526]  ffff88002c053910 ffffffff816f9258 ffff88002c053fd8 ffff88002c053970
 [ 7123.883837]  ffffffff817041c7 ffff88002a984600 ffff88002c053fd8 ffff88002c053fd8
 [ 7123.885845] Call Trace:
 [ 7123.886437]  [<ffffffff816fd400>] dump_stack+0x19/0x1b
 [ 7123.887079]  [<ffffffff816f9258>] __schedule_bug+0x59/0x68
 [ 7123.887703]  [<ffffffff817041c7>] __schedule+0x807/0x950
 [ 7123.894355]  [<ffffffffa0589c50>] ? ldlm_extent_compat_queue+0x9c0/0x9c0 [ptlrpc]
 [ 7123.895471]  [<ffffffff81704339>] schedule+0x29/0x70
 [ 7123.896124]  [<ffffffff81700972>] schedule_timeout+0x162/0x2a0
 [ 7123.896759]  [<ffffffff810879f0>] ? call_timer_fn+0x1e0/0x1e0
 [ 7123.897499]  [<ffffffffa0589ffa>] ? ldlm_process_extent_lock+0x3aa/0x5b0 [ptlrpc]
 [ 7123.904218]  [<ffffffffa01d20b3>] __cfs_fail_timeout_set+0xe3/0x170 [libcfs]
 [ 7123.904882]  [<ffffffffa0dd5dd2>] ofd_intent_policy+0x622/0xa20 [ofd]
 [ 7123.905537]  [<ffffffffa05720ea>] ldlm_lock_enqueue+0x35a/0x940 [ptlrpc]
 [ 7123.906167]  [<ffffffffa01da4b5>] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs]
 [ 7123.906771]  [<ffffffffa059b7e3>] ldlm_handle_enqueue0+0x9e3/0x1670 [ptlrpc]
 [ 7123.907409]  [<ffffffffa05c3870>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
 [ 7123.908693]  [<ffffffffa061ebd2>] tgt_enqueue+0x62/0x210 [ptlrpc]
 [ 7123.909354]  [<ffffffffa0623015>] tgt_request_handle+0x925/0x1330 [ptlrpc]
 [ 7123.909961]  [<ffffffffa05cca51>] ptlrpc_server_handle_request+0x231/0xac0 [ptlrpc]
 [ 7123.911062]  [<ffffffffa05ca608>] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc]
 [ 7123.911806]  [<ffffffffa05d07e8>] ptlrpc_main+0xa58/0x1dd0 [ptlrpc]
 [ 7123.912495]  [<ffffffff81706487>] ? _raw_spin_unlock_irq+0x27/0x50
 [ 7123.913204]  [<ffffffffa05cfd90>] ? ptlrpc_register_service+0xe90/0xe90 [ptlrpc]
 [ 7123.914491]  [<ffffffff810a2eda>] kthread+0xea/0xf0
 [ 7123.916823]  [<ffffffff810a2df0>] ? kthread_create_on_node+0x140/0x140
 [ 7123.917579]  [<ffffffff8170fbd8>] ret_from_fork+0x58/0x90
 [ 7123.928412]  [<ffffffff810a2df0>] ? kthread_create_on_node+0x140/0x140
 [ 7125.852041] LustreError: 23298:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 30f awake
 [ 7125.853859] LustreError: 23298:0:(ldlm_lock.c:1736:ldlm_lock_enqueue()) ASSERTION( *flags & 0x0100000000000000ULL ) failed: 
 [ 7125.855098] LustreError: 23298:0:(ldlm_lock.c:1736:ldlm_lock_enqueue()) LBUG
 [ 7125.855669] Pid: 23298, comm: ll_ost00_007
 [ 7125.856203] 
 Call Trace:
 [ 7125.857221]  [<ffffffffa01c97ce>] libcfs_call_trace+0x4e/0x60 [libcfs]
 [ 7125.857789]  [<ffffffffa01c985c>] lbug_with_loc+0x4c/0xb0 [libcfs]
 [ 7125.858394]  [<ffffffffa05726cd>] ldlm_lock_enqueue+0x93d/0x940 [ptlrpc]
 [ 7125.858972]  [<ffffffffa01da4b5>] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs]
 [ 7125.859575]  [<ffffffffa059b7e3>] ldlm_handle_enqueue0+0x9e3/0x1670 [ptlrpc]
 [ 7125.860207]  [<ffffffffa05c3870>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc]
 [ 7125.861354]  [<ffffffffa061ebd2>] tgt_enqueue+0x62/0x210 [ptlrpc]
 [ 7125.861915]  [<ffffffffa0623015>] tgt_request_handle+0x925/0x1330 [ptlrpc]
 [ 7125.862364]  [<ffffffffa05cca51>] ptlrpc_server_handle_request+0x231/0xac0 [ptlrpc]
 [ 7125.863214]  [<ffffffffa05ca608>] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc]
 [ 7125.864057]  [<ffffffffa05d07e8>] ptlrpc_main+0xa58/0x1dd0 [ptlrpc]
 [ 7125.865669]  [<ffffffff81706487>] ? _raw_spin_unlock_irq+0x27/0x50
 [ 7125.866698]  [<ffffffffa05cfd90>] ? ptlrpc_main+0x0/0x1dd0 [ptlrpc]
 [ 7125.867371]  [<ffffffff810a2eda>] kthread+0xea/0xf0
 [ 7125.867998]  [<ffffffff810a2df0>] ? kthread+0x0/0xf0
 [ 7125.868647]  [<ffffffff8170fbd8>] ret_from_fork+0x58/0x90
 [ 7125.869294]  [<ffffffff810a2df0>] ? kthread+0x0/0xf0
 [ 7125.869987] 
 [ 7125.870561] Kernel panic - not syncing: LBUG

Issue Links

• is related to: LU-9595 sanityn test cases 18c 29 51a remove from ALWAYS_EXCEPT
• is related to: LU-7108 Remove sanityn tests 14b, 19, 29, and 35 from the ALWAYS_EXCEPT list