[LU-7028] racer:kernel:BUG: spinlock bad magic on CPU#0

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Fix Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Major
• Reporter: Ashish Purkar (Inactive)
• Assignee: Emoly Liu
• Resolution: Fixed
• Labels: patch

Environment

Single Node Stetup, Scientific Linux (2.6.32-431-17.1.el6.x86_x64-Lustre)

• Severity: 3

Description

Running racer on master(2.7.58-22-g1d40519) with Single Node setup failed with following crash -
Message from syslogd@maximus at Aug 21 03:58:02 ...
kernel:BUG: spinlock bad magic on CPU#0, mdt00_002/3228 (Tainted: G --------------- H )
Aug 21 03:58:02 maximus kernel: BUG: spinlock bad magic on CPU#0, mdt00_002/3228 (Tainted: G --------------- H )

Message from syslogd@maximus at Aug 21 03:58:02 ...
kernel: lock: ffff88001dd76408, .magic: 00000000, .owner: <none>/-1, .owner_cpu: 0
Aug 21 03:58:02 maximus kernel: lock: ffff88001dd76408, .magic: 00000000, .owner: <none>/-1, .owner_cpu: 0
Aug 21 03:58:02 maximus kernel: Pid: 3228, comm: mdt00_002 Tainted: G--------------- H 2.6.32 #2
Aug 21 03:58:02 maximus kernel: Call Trace:
Aug 21 03:58:02 maximus kernel: [<ffffffff8129c3ca>] ? spin_bug+0xaa/0x100
Aug 21 03:58:02 maximus kernel: [<ffffffff8129c565>] ? _raw_spin_lock+0xa5/0x180

Aug 21 03:58:02 maximus kernel: [<ffffffff81173f7c>] ? kmem_cache_alloc_trace+0x20c/0x280
Aug 21 03:58:02 maximus kernel: [<ffffffff815378ee>] ? _spin_lock+0xe/0x10
Aug 21 03:58:02 maximus kernel: [<ffffffffa064ec32>] ? tgt_bitmap_chunk_alloc+0x92/0x280 [ptlrpc]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0652ce3>] ? tgt_txn_stop_cb+0x12f3/0x1320 [ptlrpc]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0a4a796>] ? ldiskfs_mark_inode_dirty+0x96/0x1f0 [ldiskfs]
Aug 21 03:58:02 maximus kernel: [<ffffffffa03d683e>] ? dt_txn_hook_stop+0x5e/0x90 [obdclass]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0aa8100>] ? osd_trans_stop+0x190/0x590 [osd_ldiskfs]
Aug 21 03:58:34 maximus kernel: LustreError: 19055:0:(file.c:184:ll_close_inode_openhandle()) lustre-clilmv-ffff880015c956b8: inode [0x200000401:0x647:0x0] mdc
close failed: rc = -13s kernel: [<ffffffffa0ccf8a6>] ? lod_xattr_set+0x156/0x3e0
Aug 21 03:58:45 maximus kernel: LustreError: 24291:0:(file.c:184:ll_close_inode_openhandle()) lustre-clilmv-ffff8800155c4860: inode [0x200000402:0x96a:0x0] mdc
close failed: rc = -13
Aug 21 03:58:56 maximus kernel: LustreError: 28956:0:(file.c:184:ll_close_inode_openhandle()) lustre-clilmv-ffff8800155c4860: inode [0x200000402:0xb36:0x0] mdc
close failed: rc = -13s kernel: [<ffffffffa03f2b70>] ? lu_ucred+0x20/0x30 [obdclass]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0cab8dc>] ? lod_trans_stop+0x2bc/0x330 [lod]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0ba97ba>] ? mdd_trans_stop+0x1a/0x1c[mdd]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0b98bd9>] ? mdd_create+0x13b9/0x1760[mdd]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0bf83b4>] ? mdt_version_save+0x84/0x1a0 [mdt]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0bfc6fc>] ? mdt_reint_create+0xbbc/0xcc0 [mdt]
Aug 21 03:58:02 maximus kernel: [<ffffffffa03f2b70>] ? lu_ucred+0x20/0x30 [obdclass]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0bda245>] ? mdt_ucred+0x15/0x20 [mdt]

Aug 21 03:58:02 maximus kernel: [<ffffffffa0bf353c>] ? mdt_root_squash+0x2c/0x3f0 [mdt]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0622072>] ? __req_capsule_get+0x162/0x6e0 [ptlrpc]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0bf76fd>] ? mdt_reint_rec+0x5d/0x200[mdt]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0be0bfb>] ? mdt_reint_internal+0x62b/0xa40 [mdt]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0be14ab>] ? mdt_reint+0x6b/0x120 [mdt]
Aug 21 03:58:02 maximus kernel: [<ffffffffa0660bf2>] ? tgt_request_handle+0xa42/0x1230 [ptlrpc]
Aug 21 03:58:02 maximus kernel: [<ffffffffa060be84>] ? ptlrpc_main+0xd74/0x1850[ptlrpc]
Aug 21 03:58:02 maximus kernel: [<ffffffffa060b110>] ? ptlrpc_main+0x0/0x1850 [ptlrpc]
Aug 21 03:58:02 maximus kernel: [<ffffffff8109b656>] ? kthread+0x96/0xa0
Aug 21 03:58:02 maximus kernel: [<ffffffff8100c24a>] ? child_rip+0xa/0x20
Aug 21 03:58:02 maximus kernel: [<ffffffff8109b5c0>] ? kthread+0x0/0xa0
Aug 21 03:58:02 maximus kernel: [<ffffffff8100c240>] ? child_rip+0x0/0x20
Aug 21 03:58:17 maximus kernel: LustreError: 11382:0:(file.c:184:ll_close_inode_openhandle()) lustre-clilmv-ffff880015c956b8: inode [0x200000401:0x32a:0x0] mdc
close failed: rc = -13
Aug 21 03:58:29 maximus kernel: LustreError: 17103:0:(file.c:184:ll_close_inode_openhandle()) lustre-clilmv-ffff880015c956b8: inode [0x200000402:0x61b:0x0] mdc
close failed: rc = -13

Issue Links

• is related to: LU-5319 Support multiple slots per client in ...

Comments

[Ashish Purkar (Inactive)]

Issue here is uninitialized spin lock -

diff --git a/lustre/target/tgt_main.c b/lustre/target/tgt_main.c
index a920716..c2b518f 100644
--- a/lustre/target/tgt_main.c
+++ b/lustre/target/tgt_main.c
@@ -80,6 +80,7 @@ int tgt_init(const struct lu_env *env, struct lu_target *lut,
                RETURN(0);

        spin_lock_init(&lut->lut_translock);
+       spin_lock_init(&lut->lut_client_bitmap_lock);

        OBD_ALLOC(lut->lut_client_bitmap, LR_MAX_CLIENTS >> 3);
        if (lut->lut_client_bitmap == NULL)

[Gerrit Updater]

Ashish Purkar ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16045
Subject: LU-7028 tgt: initialize spin lock in tgt_init()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e0b505aca053894912a514f2be3a459591bb8175

[Joseph Gmitter (Inactive)]

Hi Emoly,
Can you have a look at this patch?
Thanks.
Joe

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/16045/
Subject: LU-7028 tgt: initialize spin lock in tgt_init()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0b813eea77eff5a4753f6e862d4a93da1b0329b6

[Joseph Gmitter (Inactive)]

Landed for 2.8.