[LU-2591] race between mount/umount and lov_notify

Tue, 5 Mar 2013 23:23:54 +0000

I've found a race between mount/umount and lov_notify().
The race is caused by the following scenario

1) mount/umount: mount runs and fails to communicate with some OSTs. Then import objects are registered to a pinger list.
2) pinger : pinger succeeds to communicate with one OST
3) ptrprc_rcv : ptlrpc_rcv calls lov_set_osc_active() and holds lov_refcount.
4) mount/umount: For some reason, possibly mount fails or umount runs, ll_put_super() is called.
5) mount/umount: lov_disconnect() calls lov_del_target() and set all target->ltd_reap to actually disconnect OSCs.
6) ptlrpc_rcv : puts lov_refcount which ptlrpc_rcv has gotten at (3). If lov_refcount becomes 0 here, ptlrpc_rcv tries to disconnect all OSCs at __lov_del_target()
7) ptlrpc_rcv : some OSC's imports have still been LUSTRE_IMP_CONNECTING state, so ptlrpc_rcv thread have to wait for these imports' state to be changed to non-recovery state at ptlrpc_disconnect_import().

Now that ptlrpc_rcv thread is waiting for the import state to be changed to non-recovery state but ptlrpc_rcv is the one who is supposed to change a recovery state to a non-recovery state. So ptlrpc_rcv must hung. And that's why this node is not able to use ptlrpc_rcv thread anymore.

-------------
At first, I saw this problem in Lustre-1.8.8 and this scenario ended up only in mount/umount command hung.
But I've seen a lot more kinds of problems in Lustre-2.3.x than in Lustre-1.8.8 by my reproducer. And one of them looks like the same as ~~LU-2070~~. So I assume that ~~LU-2070~~ may be a duplicate of this case, not ~~LU-1480~~, although I'm not so sure.

Jan  8 14:00:05 rx200-088 kernel: Lustre: Lustre: Build Version: 2.1.56-gf394dce-CHANGED-2.6.32-279.5.1.el6_lustre.gb16fe80.x86_64
Jan  8 14:00:06 rx200-088 kernel: LNet: Added LNI 192.168.128.88@o2ib [8/256/0/180]
Jan  8 14:00:06 rx200-088 kernel: Lustre: MGC192.168.128.86@o2ib: Reactivating import

----- This is because of my reproducer -----
Jan  8 14:00:06 rx200-088 kernel: LustreError: 2726:0:(llite_lib.c:562:client_common_fill_super()) lustre: can't make root dentry
--------------------------------------------

Jan  8 14:00:11 rx200-088 kernel: LustreError: 2773:0:(lov_obd.c:465:lov_set_osc_active()) ===== ACTIVE WAIT =====
Jan  8 14:00:16 rx200-088 kernel: LustreError: 2753:0:(lu_object.c:1114:lu_device_fini()) ASSERTION( cfs_atomic_read(&d->ld_ref) == 0 ) failed: Refcount is 1
Jan  8 14:00:16 rx200-088 kernel: LustreError: 2773:0:(lov_obd.c:469:lov_set_osc_active()) ===== ACTIVE END =====
Jan  8 14:00:16 rx200-088 kernel: LustreError: 2773:0:(lov_obd.c:503:lov_notify()) event(2) of lustre-OST0001_UUID failed: -22
Jan  8 14:00:16 rx200-088 kernel: LustreError: 2753:0:(lu_object.c:1114:lu_device_fini()) LBUG
Jan  8 14:00:16 rx200-088 kernel: Pid: 2753, comm: obd_zombid
Jan  8 14:00:16 rx200-088 kernel:
Jan  8 14:00:16 rx200-088 kernel: Call Trace:
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa03b5905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa03b5f17>] lbug_with_loc+0x47/0xb0 [libcfs]
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa0533ecc>] lu_device_fini+0xcc/0xd0 [obdclass]
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa0948b9e>] osc_device_free+0x6e/0x220 [osc]
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa0511d8d>] class_decref+0x46d/0x590 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04e9c78>] ? class_import_destroy+0x208/0x450 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ede29>] obd_zombie_impexp_cull+0x309/0x610 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ee1f5>] obd_zombie_impexp_thread+0xc5/0x1c0 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffff81060250>] ? default_wake_function+0x0/0x20
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ee130>] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ee130>] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ee130>] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jan  8 14:00:17 rx200-088 kernel:
Jan  8 14:00:17 rx200-088 kernel: Kernel panic - not syncing: LBUG

-----------------------------

To fix the problems, I've added a new rw_semaphore into struct lov_obd to exclude lov_notify() and lov_del_target(). I think this is one of the right ways to fix the problems because I've never seen all of the problems since I applied the patch. But I'm wondering whether or not I'm allowed to add a new menber into a basic object.

I'll upload the patch soon, so could you please review it?
Thank you

Whamcloud Community JIRA

[LU-2591] race between mount/umount and lov_notify