Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
Lustre 2.3.0
-
None
-
server: lustre-master-tag-2.2.91 RHEL6 OFED
client: lustre-master-tag-2.2.91 RHEL6 OFED
-
3
-
4497
Description
From the log, racer completed while after that got a LBUG
Call Trace:
[<ffffffffa0c28905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
[<ffffffffa094be75>] osc_lock_enqueue+0x785/0x9d0 [osc]
[<ffffffffa0c39521>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa055302c>] cl_enqueue_try+0xfc/0x300 [obdclass]
[<ffffffffa09f620a>] lov_lock_enqueue+0x23a/0x830 [lov]
[<ffffffffa055302c>] cl_enqueue_try+0xfc/0x300 [obdclass]
[<ffffffffa055452d>] cl_enqueue_locked+0x6d/0x210 [obdclass]
[<ffffffffa05551ce>] cl_lock_request+0x7e/0x280 [obdclass]
[<ffffffffa055a636>] cl_io_lock+0x3d6/0x5b0 [obdclass]
[<ffffffffa055a8b2>] cl_io_loop+0xa2/0x1b0 [obdclass]
[<ffffffffa0ad793e>] ll_file_io_generic+0x42e/0x560 [lustre]
[<ffffffffa0ad884c>] ll_file_aio_write+0x13c/0x2c0 [lustre]
[<ffffffffa0ad8b39>] ll_file_write+0x169/0x2a0 [lustre]
[<ffffffff81176918>] vfs_write+0xb8/0x1a0
[<ffffffff810d4932>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff81177321>] sys_write+0x51/0x90
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
Lustre: DEBUG MARKER: == racer racer.sh test complete, duration 930 sec == 18:38:51 (1342834731)
LustreError: 11662:0:(lov_object.c:157:lov_init_sub()) Stripe is already owned by other file (0).
LustreError: 11662:0:(lov_object.c:158:lov_init_sub()) header@ffff88014e069ca0[0x0, 2, [0x100050000:0x11509:0x0] hash]{
LustreError: 11662:0:(lov_object.c:158:lov_init_sub()) ....lovsub@ffff88014e069d38[0]
LustreError: 11662:0:(lov_object.c:158:lov_init_sub()) ....osc@ffff8802d9037408id: 70921 gr: 0 idx: 5 gen: 0 kms_valid: 0 kms 0 rc: 0 force_sync: 0 min_xid: 0 size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
LustreError: 11662:0:(lov_object.c:158:lov_init_sub()) } header@ffff88014e069ca0
LustreError: 11662:0:(lov_object.c:158:lov_init_sub())
LustreError: 11662:0:(lov_object.c:160:lov_init_sub()) header@ffff88020a0d5870[0x0, 1, [0xc00073:0xd056be3f:0x0] hash]{
LustreError: 11662:0:(lov_object.c:160:lov_init_sub()) ....vvp@ffff88020a0d5908(- 0 0) inode: ffff8802279d7c78 12583027/3495345727 100644 1 1 ffff88020a0d5908 [0xc00073:0xd056be3f:0x0]
LustreError: 11662:0:(lov_object.c:160:lov_init_sub()) ....lov@ffff8802fa137e10stripes: 1:
LustreError: 11662:0:(lov_object.c:160:lov_init_sub()) header@ffff88014e069ca0[0x0, 2, [0x100050000:0x11509:0x0] hash]{
LustreError: 11662:0:(lov_object.c:160:lov_init_sub()) ....lovsub@ffff88014e069d38[0]
LustreError: 11662:0:(lov_object.c:160:lov_init_sub()) ....osc@ffff8802d9037408id: 70921 gr: 0 idx: 5 gen: 0 kms_valid: 0 kms 0 rc: 0 force_sync: 0 min_xid: 0 size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
LustreError: 11662:0:(lov_object.c:160:lov_init_sub()) } header@ffff88014e069ca0
LustreError: 11662:0:(lov_object.c:160:lov_init_sub())
LustreError: 11662:0:(lov_object.c:160:lov_init_sub()) } header@ffff88020a0d5870
LustreError: 11662:0:(lov_object.c:160:lov_init_sub()) old
LustreError: 11662:0:(lov_object.c:161:lov_init_sub()) header@ffff88020a0d5170[0x0, 1, [0x200002343:0xb501:0x0]]
LustreError: 11662:0:(lov_object.c:161:lov_init_sub()) new
LustreError: 11662:0:(lov_object.c:327:lov_fini_raid0()) ASSERTION( cfs_atomic_read(&lov->lo_lsm->lsm_refc) == 1 ) failed: actual 2 proc ffff880312bca0c0.
LustreError: 11662:0:(lov_object.c:327:lov_fini_raid0()) LBUG
Pid: 11662, comm: ll_sa_11661
Call Trace:
[<ffffffffa0c28905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Message from [<ffffffffa0c28f17>] lbug_with_loc+0x47/0xb0 [libcfs]
syslogd@client-4 [<ffffffffa09ee80e>] lov_fini_raid0+0x2ee/0x2f0 [lov]
at Jul 20 18:38 [<ffffffffa09f1539>] lov_object_init+0x129/0x1c0 [lov]
:52 ...
kerne [<ffffffffa05400e5>] lu_object_alloc+0xd5/0x310 [obdclass]
l:LustreError: 1 [<ffffffffa05408a9>] lu_object_find_at+0x139/0x450 [obdclass]
1662:0:(lov_obje [<ffffffffa0b225af>] ? cl_file_inode_init+0x5f/0x360 [lustre]
ct.c:327:lov_fin [<ffffffffa0c39521>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
i_raid0()) ASSER [<ffffffffa0540bff>] lu_object_find_slice+0x1f/0x80 [obdclass]
TION( cfs_atomic [<ffffffffa0547602>] cl_object_find+0x42/0xb0 [obdclass]
_read(&lov->lo_l [<ffffffffa0b2276f>] cl_file_inode_init+0x21f/0x360 [lustre]
sm->lsm_refc) == [<ffffffffa0aefea2>] ll_update_inode+0x112/0xe60 [lustre]
1 ) failed: act [<ffffffffa0c39521>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
ual 2 proc ffff8 [<ffffffffa0af1868>] ll_read_inode2+0x88/0x440 [lustre]
80312bca0c0.
[<ffffffffa0b07dc3>] ll_iget+0x1a3/0x2a0 [lustre]
Message from [<ffffffffa0af10d1>] ll_prep_inode+0x4e1/0xbf0 [lustre]
syslogd@client-4 [<ffffffffa0b1c417>] do_statahead_interpret+0x347/0xde0 [lustre]
at Jul 20 18:38 [<ffffffffa0b2077a>] ll_statahead_thread+0x27a/0xf60 [lustre]
:52 ...
kerne [<ffffffff8105ea30>] ? default_wake_function+0x0/0x20
l:LustreError: 1 [<ffffffffa0b20500>] ? ll_statahead_thread+0x0/0xf60 [lustre]
1662:0:(lov_obje [<ffffffff8100c14a>] child_rip+0xa/0x20
ct.c:327:lov_fin [<ffffffffa0b20500>] ? ll_statahead_thread+0x0/0xf60 [lustre]
[<ffffffffa0b20500>] ? ll_statahead_thread+0x0/0xf60 [lustre]
[<ffffffff8100c140>] ? child_rip+0x0/0x20
Kernel panic - not syncing: LBUG
Pid: 11662, comm: ll_sa_11661 Not tainted 2.6.32-220.17.1.el6.x86_64 #1
Call Trace:
[<ffffffff814ec98a>] ? panic+0x78/0x143
[<ffffffffa0c28f6b>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
[<ffffffffa09ee80e>] ? lov_fini_raid0+0x2ee/0x2f0 [lov]
Message from sy [<ffffffffa09f1539>] ? lov_object_init+0x129/0x1c0 [lov]
slogd@client-4 a [<ffffffffa05400e5>] ? lu_object_alloc+0xd5/0x310 [obdclass]
t Jul 20 18:38:5 [<ffffffffa05408a9>] ? lu_object_find_at+0x139/0x450 [obdclass]
2 ...
kernel: [<ffffffffa0b225af>] ? cl_file_inode_init+0x5f/0x360 [lustre]
Kernel panic - n [<ffffffffa0c39521>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
ot syncing: LBUG [<ffffffffa0540bff>] ? lu_object_find_slice+0x1f/0x80 [obdclass]
[<ffffffffa0547602>] ? cl_object_find+0x42/0xb0 [obdclass]
[<ffffffffa0b2276f>] ? cl_file_inode_init+0x21f/0x360 [lustre]
[<ffffffffa0aefea2>] ? ll_update_inode+0x112/0xe60 [lustre]
[<ffffffffa0c39521>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[<ffffffffa0af1868>] ? ll_read_inode2+0x88/0x440 [lustre]
[<ffffffffa0b07dc3>] ? ll_iget+0x1a3/0x2a0 [lustre]
[<ffffffffa0af10d1>] ? ll_prep_inode+0x4e1/0xbf0 [lustre]
[<ffffffffa0b1c417>] ? do_statahead_interpret+0x347/0xde0 [lustre]
[<ffffffffa0b2077a>] ? ll_statahead_thread+0x27a/0xf60 [lustre]
[<ffffffff8105ea30>] ? default_wake_function+0x0/0x20
[<ffffffffa0b20500>] ? ll_statahead_thread+0x0/0xf60 [lustre]
[<ffffffff8100c14a>] ? child_rip+0xa/0x20
[<ffffffffa0b20500>] ? ll_statahead_thread+0x0/0xf60 [lustre]
[<ffffffffa0b20500>] ? ll_statahead_thread+0x0/0xf60 [lustre]
[<ffffffff8100c140>] ? child_rip+0x0/0x20
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu