[#LU-948] Client recovery hang

[LU-948] Client recovery hang Created: 20/Dec/11 Updated: 16/Aug/16 Resolved: 16/Aug/16
Status:	Closed
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.1.0
Fix Version/s:	Lustre 2.2.0, Lustre 2.1.3

Type:

Bug

Priority:

Minor

Reporter:

Christopher Morrone

Assignee:

Jinshan Xiong (Inactive)

Resolution:

Won't Fix

Votes:

Labels:

llnl

Attachments:

hype320_bt_pages.txt.bz2

seqlac2_console.txt

Issue Links:

Duplicate
is duplicated by	~~LU-1066~~	Test failure on test suite replay-sin...	Resolved
Related
is related to	~~LU-1059~~	vvp_page_unmap()) ASSERTION(PageLocke...	Resolved

Severity:

Rank (Obsolete):

4759

Description

When I rebooted two OSS to put a patch for bug ~~LU-874~~ on the servers, quite a few of the clients have appear to have gotten deadlocked in recovery. Here's a backtrace of ptlrpcd-rcv on on client:

crash> bt 5077
PID: 5077   TASK: ffff88082da834c0  CPU: 8   COMMAND: "ptlrpcd-rcv"
 #0 [ffff88082da85430] schedule at ffffffff814ee3b2
 #1 [ffff88082da854f8] io_schedule at ffffffff814eeba3
 #2 [ffff88082da85518] sync_page at ffffffff81110fbd
 #3 [ffff88082da85528] __wait_on_bit_lock at ffffffff814ef40a
 #4 [ffff88082da85578] __lock_page at ffffffff81110f57
 #5 [ffff88082da855d8] vvp_page_own at ffffffffa093bf6a [lustre]
 #6 [ffff88082da855f8] cl_page_own0 at ffffffffa0601d3b [obdclass]
 #7 [ffff88082da85678] cl_page_own at ffffffffa0601fa0 [obdclass]
 #8 [ffff88082da85688] cl_page_gang_lookup at ffffffffa0603bb7 [obdclass]
 #9 [ffff88082da85758] cl_lock_page_out at ffffffffa06096fc [obdclass]
#10 [ffff88082da85808] osc_lock_flush at ffffffffa0858e8f [osc]
#11 [ffff88082da85858] osc_lock_cancel at ffffffffa0858f2a [osc]
#12 [ffff88082da858d8] cl_lock_cancel0 at ffffffffa0604665 [obdclass]
#13 [ffff88082da85928] cl_lock_cancel at ffffffffa06051ab [obdclass]
#14 [ffff88082da85968] osc_ldlm_blocking_ast at ffffffffa0859cf8 [osc]
#15 [ffff88082da859f8] ldlm_cancel_callback at ffffffffa06a1ba3 [ptlrpc]
#16 [ffff88082da85a18] ldlm_lock_cancel at ffffffffa06a1c89 [ptlrpc]
#17 [ffff88082da85a58] ldlm_cli_cancel_list_local at ffffffffa06bede8 [ptlrpc]
#18 [ffff88082da85ae8] ldlm_cancel_lru_local at ffffffffa06bf255 [ptlrpc]
#19 [ffff88082da85b08] ldlm_replay_locks at ffffffffa06bf385 [ptlrpc]
#20 [ffff88082da85bb8] ptlrpc_import_recovery_state_machine at ffffffffa070ceea [ptlrpc]
#21 [ffff88082da85c38] ptlrpc_connect_interpret at ffffffffa070db38 [ptlrpc]
#22 [ffff88082da85d08] ptlrpc_check_set at ffffffffa06dd870 [ptlrpc]
#23 [ffff88082da85de8] ptlrpcd_check at ffffffffa07113b8 [ptlrpc]
#24 [ffff88082da85e48] ptlrpcd at ffffffffa071175b [ptlrpc]
#25 [ffff88082da85f48] kernel_thread at ffffffff8100c14a

I will need to do more investigation, but thats a start.

Comments

Comment by Christopher Morrone [ 20/Dec/11 ]

Backtrace of all tasks on a hung client, and a dump_pages.

Comment by Peter Jones [ 20/Dec/11 ]

Jinshan is looking into this

Comment by Jinshan Xiong (Inactive) [ 20/Dec/11 ]

Do you have a chance to dump lustre log at clients?

Comment by Christopher Morrone [ 20/Dec/11 ]

No, unfortunately someone rebooted the cluster out from under me. That was all that I could grab before the nodes went dead.

Comment by Jinshan Xiong (Inactive) [ 21/Dec/11 ]

I've known the root cause of this problem. If there is a BRW RPC in the resending list, canceling the ldlm lock will cause deadlock since it will try to hold page lock. A workaround to fix this problem is by disabling ldlm_cancel_unused_locks_before_replay via procfs.

I'm working on a fix.

Comment by Christopher Morrone [ 10/Jan/12 ]

With ~~LU-874~~ much improved, this is our next concern for 2.1. Have you had any time to look at the fix for this?

Comment by Jinshan Xiong (Inactive) [ 10/Jan/12 ]

I think this issue can be fixed by reverting commit 6fd5e00ff03d41b427eec5d70efaef4bbdd8d59c which was added in bug 16774 to address client replaying lots of unused lock during recovery.

Since this issue was filed and even implemented by you guys, can you please tell me what's the side effect for clients to replay unused lock during recovery?

Comment by Christopher Morrone [ 10/Jan/12 ]

We only implemented some of the 1.8 version. The CLIO version for 2.X doesn't look at all familiar to me.

In production, clients will need to replay tens of thousands of locks, which completely overwhelms the servers. Since most of those locks are completely usused, it is better to drop the unused locks rather than replay them. If they are needed again in the future, the load to recreate them on demand is easier to deal with than the flood of lock replays at recovery time.

At the time we could even see the problem with just one or a few clients. If you did something like a linux kernel compilation out of lustre, you will wind up with tens of thousands of locks on just that one node.

I don't think we really want to abandon this ability.

Comment by Jinshan Xiong (Inactive) [ 11/Jan/12 ]

patch is at: http://review.whamcloud.com/1955