|
Fan Yong
Could you please advise on this issue?
Thanks
Peter
|
|
Here is the LFSCK logs on the lola-8:
That means both the layout LFSCK assistant thread and the namespace LFSCK assistant thread have exited finally, although there were a lot failures when communicated with remote peers. So what Cliff saw about the LFSCK on the MDT was during the connection issues with lola-4. The MDT0000 needs to talks with the OSTs on lola-4 for layout LFSCK, since there were connection rouble with the lola-4, the LFSCK will retry until recovered or manually stopped or exited. So the LFSCK behaviour is in expectation. What we need to make clear is why the connection between the lola-8 and lola-4 were repeatedly down and up.
|
|
The logs on the lola-4 show that at the beginning, the txg_quiesce_thread was blocked for some unknown reason.
The txg_quiesce_thread() is used for handling the ZFS transactions. Such hung caused a lot of subsequent transactions blocked. For example:
These OST RPC services thread hung caused related MDT/client RPC timeout, then triggered the reconnection from the MDT/client.
For MDT, after reconnected, it sent orphan cleanup RPC to the OST for sync the last_id. Unfortunately, related RPC service thread was blocked by the mutex ofd_seq::os_create_lock that was held by another RPC service thread which was blocked by the transaction trouble. So there were a serious of blocking. And then these blocked orphan cleanup RPC timeout and trigger more and more reconnection repeatedly.
So the root is why the txg_quiesce_thread was blocked.
|
|
There are no detailed Lustre debug logs for the txg_quiesce_thread hung. By simply comparing the stack trace, there are some similar tickets on Jira, such as LU-5165.
Add Alex for more suggestion.
|
|
looks like a lock/tx ordering issue.
here we take the object's lock before TX
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa038c81d>] dmu_tx_wait+0x21d/0x400 [zfs]
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa03d1ef2>] ? txg_kick+0x92/0xd0 [zfs]
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa038cbf1>] dmu_tx_assign+0xa1/0x570 [zfs]
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa0f06d7d>] osd_trans_start+0xed/0x430 [osd_zfs]
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa102156c>] ofd_trans_start+0x7c/0x100 [ofd]
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa1023980>] ofd_object_destroy+0x3d0/0x8e0 [ofd]
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa101d2cd>] ofd_destroy_by_fid+0x35d/0x620 [ofd]
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa098bab0>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa098db10>] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc]
Nov 4 09:31:32 lola-4 kernel: [<ffffffffa10176a2>] ofd_destroy_hdl+0x2f2/0xb60 [ofd]
and here we do with TX started:
Nov 4 09:32:15 lola-4 kernel: Pid: 33274, comm: ll_ost_out01_00
Nov 4 09:32:15 lola-4 kernel:
Nov 4 09:32:15 lola-4 kernel: Call Trace:
Nov 4 09:32:15 lola-4 kernel: [<ffffffffa03a22ce>] ? dsl_dir_tempreserve_impl+0x2ae/0x330 [zfs]
Nov 4 09:32:15 lola-4 kernel: [<ffffffff8153cc55>] rwsem_down_failed_common+0x95/0x1d0
Nov 4 09:32:15 lola-4 kernel: [<ffffffff8153cdb3>] rwsem_down_write_failed+0x23/0x30
Nov 4 09:32:15 lola-4 kernel: [<ffffffff8129e923>] call_rwsem_down_write_failed+0x13/0x20
Nov 4 09:32:15 lola-4 kernel: [<ffffffff8153c2b2>] ? down_write+0x32/0x40
Nov 4 09:32:15 lola-4 kernel: [<ffffffffa0f0f021>] osd_object_write_lock+0x61/0x70 [osd_zfs]
Nov 4 09:32:15 lola-4 kernel: [<ffffffffa0a2bbf9>] out_tx_attr_set_exec+0x69/0x480 [ptlrpc]
Nov 4 09:32:15 lola-4 kernel: [<ffffffffa0a2296a>] out_tx_end+0xda/0x5d0 [ptlrpc]
Nov 4 09:32:15 lola-4 kernel: [<ffffffffa0a282b9>] out_handle+0x11d9/0x18d0 [ptlrpc]
|
|
As I remember, there seemed some discussion about the difference between MDT and OST operations for transaction and lock order. But I cannot remember the detail. Currently, the MDT side modification is start transaction before lock the object, but the OST side is the reverse order. Because the OUT handler is shared by both sides, there always be some potential deadlock. Is there any special reason for that? Can we unify them?
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: http://review.whamcloud.com/23689
Subject: LU-8806 target: lock ostobj before transaction start
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 28bf0565e025ca20c72fb8d813274c142ed273fe
|
|
We hit this issue with 2.7 (IEEL3) code.
OST deadlocked with lfsck operation.
Stack :
__schedule
schedule
rwsem_down_write_failed
call_rwsem_down_write_failed
osd_object_write_lock
out_tx_attr_set_exec
out_tx_end
out_handle
tgt_request_handle
ptlrpc_server_handle_request
ptlrpc_main
kthread
ret_from_fork
Progs: 2213 "ll_ost_out00_00
Any plans to fix it?
|
|
Alexey
This issue is presently targeted for the 2.11 release and a draft patch of one possible patch exists already. Feel free to provide an alternative approach or feedback on any testing you conduct on the existing prototype.
Peter
|
|
Peter,
i see no updates for this patch for year. If you talk about an Alex patch, it's just a part of this solution and i will add to watchers to it.
As about alternative solution - i don't see any good for now. It just a notification - it have a 2.7,2.8, 2.9 affected.
Not about 2.10/2.11.
|
|
Peter,
You have say this is targeted for 2.11 but i don't see a progress in ticket and lfsck broken continuously.
|
|
Alexey, the https://review.whamcloud.com/23689 patch points at https://review.whamcloud.com/27488 and 23689 has been abandoned. If you are having problems with this issue, then we welcome you to review and test that patch as you said you would do a few months ago, or submit a patch of your own. That is especially useful if this code is broken continuously for you, since it means you have a good reproducer and can verify if the fix works. I would also welcome an alternative patch based on the old ext3 delete thread, that pushes the extents/blocks to be truncated off to another temporary inode during the main transaction, and then does the actual truncate asynchronously. That would speed up all ext4 truncate/unlink operations, and not require changes to Lustre.
|
|
Closing this as a duplicate of LU-10048, which landed a patch for 2.12.
|
Generated at Sat Feb 10 02:20:42 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
lola-4.syslog.txt
lola-8.lustre-log.txt.gz