Details
-
Bug
-
Resolution: Fixed
-
Major
-
Lustre 2.8.0
-
None
-
CentOS 7.2
Linux dmds1 3.10.0-327.3.1.el7_lustre.x86_64 #1 SMP Thu Feb 18 10:53:23 PST 2016 x86_64 x86_64 x86_64 GNU/Linux
-
2
-
9223372036854775807
Description
We have two lustre clusters (all machines CentOS 7.2). Each of the clusters has a lustre filesystem mounted on a CentOS 7.2 server which exports the filesystem via NFS.
The start of our issue may have coincided with a system downtime for hardware maintenance on the 'delta' cluster.
The 'delta' cluster lustre filesystem will mount properly on the 'vanlustre4' NFS server, but clients intermittently cannot access the export via autofs mounts and if they can it is after long delays. Manually mounting the NFS export on a client machine will work but there is an incredibly long delay (~ 1-5 minutes), which I assume is why autofs has issues mounting.
We also attempted to mount the 'delta' lustre filesystem on the 'vanlustre3' server which exports a lustre filesystem from the 'echo' cluster, but encounter the same long delays so it seems reasonable this is not an NFS server issue.
On the MDS server for the 'delta' cluster 'dmds1' kernel logs I see reconnecting messages relating to the NFS server vanlustre4 (10.23.22.114). As well as LustreError ldlm_lib.c:3169:target_bulk_io(). I've attached the 'Lustre' logs from syslog.
I've also attached an strace of a client machine manually mounting the 'delta' lustre filesystem export from 'vanlustre4'.
On 'vanlustre4' messages shows multiple auth attempts from my client during the above manual mount which did eventually succeed:
Dec 13 10:19:59 vanlustre4 rpc.mountd[2807]: authenticated mount request from 10.23.32.109:877 for /user_data (/user_data)
Dec 13 10:20:04 vanlustre4 rpc.mountd[2805]: authenticated mount request from 10.23.32.109:877 for /user_data (/user_data)
Dec 13 10:20:14 vanlustre4 rpc.mountd[2812]: authenticated mount request from 10.23.32.109:877 for /user_data (/user_data)
Dec 13 10:20:34 vanlustre4 rpc.mountd[2808]: authenticated mount request from 10.23.32.109:766 for /user_data (/user_data)
Dec 13 10:20:58 vanlustre4 rpc.mountd[2809]: authenticated mount request from 10.23.32.109:766 for /user_data (/user_data)
Dec 13 10:20:58 vanlustre4 rpc.mountd[2809]: authenticated mount request from 10.23.32.109:766 for /user_data (/user_data)
Any suggestions on how to further troubleshoot this issue would be appreciated.
Attachments
Activity
Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/28321
Subject: LU-8935 ptlrpc: missing barrier before wake_up
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: a40516411152326101a7f19ac81b8d2b2429c509
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/26583/
Subject: LU-8935 ptlrpc: missing barrier before wake_up
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 33033c27aae361069877d56d44714097a208aa76
It looks like ptlrpc_client_wake_req() misses a memory barrier, which may cause ptlrpc_resend_req() wake up ptlrpc_send_rpc -> ptlrpc_register_bulk, while the latter doesn't see rq_resend set.
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/26583
Subject: LU-8935 ptlrpc: missing barrier before wake_up
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1965158cd0d0e4a0d6c1f82474e99332d963ec26
So only 55 seconds since the request was sent.
Can you increase your at_min setting to say 80 seconds and see if the frequent reconnects go away along with the Reconnet on bulk READ messages on the mds?
You need to do this on all the nodes. The downside is the detection of the dead nodes would take longer time, but I just want to ensure it's just slow MDS or something in the network that causes this.
You don't need to have any special extra debugging enabled when doing this.
Also did the lazystatfs option help with the delay?
Not sure about a specific rsync job crashing, I believe there are a few that crash. There are quite a few (hundreds) clients backing up over rsync to the NFS server.
On NFS server (PST timestamps):
Dec 22 02:03:05 vanlustre4 kernel: Lustre: 20662:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1482400977/real 1482400977] req@ffff880280affb00 x1554362320709380/t0(0) o36->delta-MDT0000-mdc-ffff880fe594d800@10.23.22.90@tcp:12/10 lens 608/1016 e 0 to 1 dl 1482400984 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Dec 22 02:03:05 vanlustre4 kernel: Lustre: 20662:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 82 previous similar messages
On MDS dmds1 (PST):
Dec 22 00:01:59 dmds1 kernel: Lustre: delta-MDT0000: Connection restored to 5d0ddfdb-65fb-bb1d-85f6-2509c2d8bc16 (at 10.23.22.114@tcp) Dec 22 00:02:06 dmds1 kernel: Lustre: delta-MDT0000: Client 89fa1c40-896d-2b3e-6ccd-c237c9d4f145 (at 10.23.22.114@tcp) reconnecting Dec 22 00:02:06 dmds1 kernel: LustreError: 22492:0:(ldlm_lib.c:3169:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff881c9e367200 x1554361449174764/t0(0) o37->89fa1c40-896d-2b3e-6ccd-c237c9d4f145@10.23.22.114@tcp:307/0 lens 568/440 e 0 to 0 dl 1482393732 ref 1 fl Interpret:/0/0 rc 0/0 Dec 22 00:02:06 dmds1 kernel: Lustre: delta-MDT0000: Connection restored to 5d0ddfdb-65fb-bb1d-85f6-2509c2d8bc16 (at 10.23.22.114@tcp) Dec 22 00:02:13 dmds1 kernel: Lustre: delta-MDT0000: Client 89fa1c40-896d-2b3e-6ccd-c237c9d4f145 (at 10.23.22.114@tcp) reconnecting Dec 22 00:02:13 dmds1 kernel: Lustre: delta-MDT0000: Connection restored to 5d0ddfdb-65fb-bb1d-85f6-2509c2d8bc16 (at 10.23.22.114@tcp)
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/28321/
Subject:
LU-8935ptlrpc: missing barrier before wake_upProject: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 0fe9e8f802f873b89283de34ab22909759235dff