[LU-2591] race between mount/umount and lov_notify

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.5, Lustre 1.8.8
• Fix Version/s: Lustre 2.4.0, Lustre 2.1.5
• Type: Bug
• Priority: Blocker
• Reporter: Hiroya Nozaki
• Assignee: Jian Yu
• Resolution: Fixed
• Labels: LB, patch
• Severity: 3

Description

I've found a race between mount/umount and lov_notify().
The race is caused by the following scenario

1) mount/umount: mount runs and fails to communicate with some OSTs. Then import objects are registered to a pinger list.
2) pinger : pinger succeeds to communicate with one OST
3) ptrprc_rcv : ptlrpc_rcv calls lov_set_osc_active() and holds lov_refcount.
4) mount/umount: For some reason, possibly mount fails or umount runs, ll_put_super() is called.
5) mount/umount: lov_disconnect() calls lov_del_target() and set all target->ltd_reap to actually disconnect OSCs.
6) ptlrpc_rcv : puts lov_refcount which ptlrpc_rcv has gotten at (3). If lov_refcount becomes 0 here, ptlrpc_rcv tries to disconnect all OSCs at __lov_del_target()
7) ptlrpc_rcv : some OSC's imports have still been LUSTRE_IMP_CONNECTING state, so ptlrpc_rcv thread have to wait for these imports' state to be changed to non-recovery state at ptlrpc_disconnect_import().

Now that ptlrpc_rcv thread is waiting for the import state to be changed to non-recovery state but ptlrpc_rcv is the one who is supposed to change a recovery state to a non-recovery state. So ptlrpc_rcv must hung. And that's why this node is not able to use ptlrpc_rcv thread anymore.

-------------
At first, I saw this problem in Lustre-1.8.8 and this scenario ended up only in mount/umount command hung.
But I've seen a lot more kinds of problems in Lustre-2.3.x than in Lustre-1.8.8 by my reproducer. And one of them looks like the same as LU-2070. So I assume that LU-2070 may be a duplicate of this case, not LU-1480, although I'm not so sure.

Jan  8 14:00:05 rx200-088 kernel: Lustre: Lustre: Build Version: 2.1.56-gf394dce-CHANGED-2.6.32-279.5.1.el6_lustre.gb16fe80.x86_64
Jan  8 14:00:06 rx200-088 kernel: LNet: Added LNI 192.168.128.88@o2ib [8/256/0/180]
Jan  8 14:00:06 rx200-088 kernel: Lustre: MGC192.168.128.86@o2ib: Reactivating import

----- This is because of my reproducer -----
Jan  8 14:00:06 rx200-088 kernel: LustreError: 2726:0:(llite_lib.c:562:client_common_fill_super()) lustre: can't make root dentry
--------------------------------------------

Jan  8 14:00:11 rx200-088 kernel: LustreError: 2773:0:(lov_obd.c:465:lov_set_osc_active()) ===== ACTIVE WAIT =====
Jan  8 14:00:16 rx200-088 kernel: LustreError: 2753:0:(lu_object.c:1114:lu_device_fini()) ASSERTION( cfs_atomic_read(&d->ld_ref) == 0 ) failed: Refcount is 1
Jan  8 14:00:16 rx200-088 kernel: LustreError: 2773:0:(lov_obd.c:469:lov_set_osc_active()) ===== ACTIVE END =====
Jan  8 14:00:16 rx200-088 kernel: LustreError: 2773:0:(lov_obd.c:503:lov_notify()) event(2) of lustre-OST0001_UUID failed: -22
Jan  8 14:00:16 rx200-088 kernel: LustreError: 2753:0:(lu_object.c:1114:lu_device_fini()) LBUG
Jan  8 14:00:16 rx200-088 kernel: Pid: 2753, comm: obd_zombid
Jan  8 14:00:16 rx200-088 kernel:
Jan  8 14:00:16 rx200-088 kernel: Call Trace:
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa03b5905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa03b5f17>] lbug_with_loc+0x47/0xb0 [libcfs]
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa0533ecc>] lu_device_fini+0xcc/0xd0 [obdclass]
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa0948b9e>] osc_device_free+0x6e/0x220 [osc]
Jan  8 14:00:16 rx200-088 kernel: [<ffffffffa0511d8d>] class_decref+0x46d/0x590 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04e9c78>] ? class_import_destroy+0x208/0x450 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ede29>] obd_zombie_impexp_cull+0x309/0x610 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ee1f5>] obd_zombie_impexp_thread+0xc5/0x1c0 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffff81060250>] ? default_wake_function+0x0/0x20
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ee130>] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ee130>] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffffa04ee130>] ? obd_zombie_impexp_thread+0x0/0x1c0 [obdclass]
Jan  8 14:00:17 rx200-088 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jan  8 14:00:17 rx200-088 kernel:
Jan  8 14:00:17 rx200-088 kernel: Kernel panic - not syncing: LBUG

-----------------------------

To fix the problems, I've added a new rw_semaphore into struct lov_obd to exclude lov_notify() and lov_del_target(). I think this is one of the right ways to fix the problems because I've never seen all of the problems since I applied the patch. But I'm wondering whether or not I'm allowed to add a new menber into a basic object.

I'll upload the patch soon, so could you please review it?
Thank you

Comments

[Hiroya Nozaki]

patch for the master branch
http://review.whamcloud.com/4979

[Jian Yu]

Hi Hiroya,

Are you going to port the patch to Lustre b2_1 and b1_8 branches?

[Hiroya Nozaki]

Hi Jian.
OK, I'll make patches for b2_1 ande b1_8 too. I'll upload them in a week. Please wait for a while.

[Hiroya Nozaki]

patch for b1_8
http://review.whamcloud.com/#change,5522

[Peter Jones]

Landed for 2.4

[Hiroya Nozaki]

Hi, Jian.
I cannot upload the patch for b2_1 because of LU-2448. Could you tell me a workaround if it exists ?

[Jian Yu]

I cannot upload the patch for b2_1 because of LU-2448. Could you tell me a workaround if it exists ?

Hi Hiroya,
The patch for LU-2448 has been landed on b2_1 branch. Could you please rebase your local b2_1 branch on the latest one?

[Hiroya Nozaki]

Thank you for your advice, Jian.
Here is the patch for b2_1.
http://review.whamcloud.com/#change,5527

[Jian Yu]

Thank you, Hiroya.