[LU-13500] Client gets evicted - nfsd non-standard errorno -108

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Campbell Mcleay (Inactive)

Description

Client is getting evicted by MDT as soon as nfsd service started on the client.

client (golf1) kernel version : 3.10.0-1062.1.1.el7_lustre.x86_64
client (golf1) lustre version : lustre-2.12.3-1.el7.x86_64

mds (gmds1) kernel version : 3.10.0-1062.1.1.el7_lustre.x86_64
mds (gmds1) lustre version : lustre-2.12.3-1.el7.x86_64

oss (goss1-goss6) kernel version : 3.10.0-1062.1.1.el7_lustre.x86_64
oss (goss1-goss6) lustre version : lustre-2.12.3-1.el7.x86_64

/etc/exports on golf1 :

/user_data       10.25.0.0/16(fsid=123456789,rw,anonuid=0,insecure,no_subtree_check,insecure_locks,async)

Apr 30 14:03:07 golf1 kernel: LustreError: 11-0: golf-MDT0000-mdc-ffff973bf6409800: operation ldlm_enqueue to node 10.25.22.90@tcp failed: rc = -107
Apr 30 14:03:07 golf1 kernel: Lustre: golf-MDT0000-mdc-ffff973bf6409800: Connection to golf-MDT0000 (at 10.25.22.90@tcp) was lost; in progress operations using this service will wait for recovery to complete
Apr 30 14:03:07 golf1 kernel: LustreError: Skipped 8 previous similar messages
Apr 30 14:03:07 golf1 kernel: LustreError: 167-0: golf-MDT0000-mdc-ffff973bf6409800: This client was evicted by golf-MDT0000; in progress operations using this service will fail.
Apr 30 14:03:07 golf1 kernel: LustreError: 25491:0:(file.c:4339:ll_inode_revalidate_fini()) golf: revalidate FID [0x20004884e:0x16:0x0] error: rc = -5
Apr 30 14:03:07 golf1 kernel: ------------[ cut here ]------------
Apr 30 14:03:07 golf1 kernel: WARNING: CPU: 26 PID: 25600 at fs/nfsd/nfsproc.c:805 nfserrno+0x58/0x70 [nfsd]
Apr 30 14:03:07 golf1 kernel: LustreError: 25579:0:(file.c:216:ll_close_inode_openhandle()) golf-clilmv-ffff973bf6409800: inode [0x20004884e:0x15:0x0] mdc close failed: rc = -108
Apr 30 14:03:07 golf1 kernel: ------------[ cut here ]------------
Apr 30 14:03:07 golf1 kernel: ------------[ cut here ]------------
Apr 30 14:03:07 golf1 kernel: nfsd: non-standard errno: -108
Apr 30 14:03:07 golf1 kernel: ------------[ cut here ]------------
Apr 30 14:03:07 golf1 kernel: WARNING: CPU: 54 PID: 25602 at fs/nfsd/nfsproc.c:805 nfserrno+0x58/0x70 [nfsd]
Apr 30 14:03:07 golf1 kernel: LustreError: 25579:0:(file.c:216:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Apr 30 14:03:07 golf1 kernel: WARNING: CPU: 24 PID: 25601 at fs/nfsd/nfsproc.c:805 nfserrno+0x58/0x70 [nfsd]
Apr 30 14:03:07 golf1 kernel: WARNING: CPU: 9 PID: 25505 at fs/nfsd/nfsproc.c:805 nfserrno+0x58/0x70 [nfsd]

Issue Links

• is related to: LU-13692 MDS slow/hung threads at mdt_object_local_lock
• is related to: LU-13054 MDS kernel BUG at ldiskfs/htree_lock.c:429!

Comments

[Oleg Drokin]

these ldlm settings look fine to me if they work for you. You can shorten the max age if you want depending on your workload.

I can't comment on the "svc_rpc_per_connection_limit" parameter as I am not familiar with that part of the NFS stack.

[Campbell Mcleay (Inactive)]

Any updates here ?

[Campbell Mcleay (Inactive)]

HI Oleg,

We have modified the config for limit the number of nfsd threads a single client can use on NFS gateways which helped to avoid the further client evictions. We are not seeing any stack traces and the backups are running smoothly so far. 

/sys/module/sunrpc/parameters/svc_rpc_per_connection_limit = 1

Also we are having the below lru size configs are in place too on the clients

lctl set_param -P ldlm.namespaces.*.lru_size=10000
lctl set_param -P ldlm.namespaces.*.lru_max_age=600000

Please check and suggest .

[Campbell Mcleay (Inactive)]

Hi Oleg,

Just some feedback on this setting: since the change has been made, there have been no stack traces, and the backups that run on that node have not been finishing early (so far). We'll see what happens when they finish, but looking much better so far!

Thanks,

Campbell

[Oleg Drokin]

So I wonder if reducing amount of mdt locks would at least help you temporarily since the problem is clearly related to parallel cancels.

lctl set_param ldlm.namespaces.*MDT*mdc*.lru_size=100

can you please run this on your nfs export node(s)? The setting is not permanent and would reset on node reboot/lustre client remount.

[Oleg Drokin]

Sorry that I have no immediate answers to this, I am still thinking about the whole thing.

[Campbell Mcleay (Inactive)]

Hello Team,

It would be really good if you can provide a solution for the same at the earliest as this is really affecting the production backup which is really putting us in a panic situations. Requesting you to check this on priority please. 

[Campbell Mcleay (Inactive)]

Any findings here please ?

[Campbell Mcleay (Inactive)]

After downgrading, back to 2.12.4, the lustre filesystem was hanging on the client (the other client we have was fine). It would work intermittently but mostly not (the other client was fine during this time). Is this due to lock recovery? I tried mounting with 'abort_recov', but same issue. It eventually resolved itself however.

There's still a large number of stack traces in the MDS log, e.g.:

Nov 16 02:58:40 hmds1 kernel: LNet: Service thread pid 16267 was inactive for 278.13s. The thread might be hung, or it might only be slow and will res
ume later. Dumping the stack trace for debugging purposes:
Nov 16 02:58:40 hmds1 kernel: LNet: Skipped 4 previous similar messages
Nov 16 02:58:40 hmds1 kernel: Pid: 16267, comm: mdt01_051 3.10.0-1062.18.1.el7_lustre.x86_64 #1 SMP Mon Jun 8 13:47:48 UTC 2020
Nov 16 02:58:40 hmds1 kernel: Call Trace:
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc12b8070>] ldlm_completion_ast+0x430/0x860 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc12ba0a1>] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc15a817b>] mdt_rename_lock+0x24b/0x4b0 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc15aa350>] mdt_reint_rename+0x2c0/0x2900 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc15b31b3>] mdt_reint_rec+0x83/0x210 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc158f383>] mdt_reint_internal+0x6e3/0xaf0 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc159b0f7>] mdt_reint+0x67/0x140 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc1356e8a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc12fb83b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc12ff1a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffff8dac6321>] kthread+0xd1/0xe0
Nov 16 02:58:40 hmds1 kernel: [<ffffffff8e18ed37>] ret_from_fork_nospec_end+0x0/0x39
Nov 16 02:58:40 hmds1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Nov 16 02:58:40 hmds1 kernel: LustreError: dumping log to /tmp/lustre-log.1605475720.16267

We've uploaded a log (5.1GB) from the MDS

[Campbell Mcleay (Inactive)]

nfs exports stopped working. Couldn't see anything in the client log.
A restart of nfs did not fix the issue. So we had to downgrade it back to 2.12.4.

[Campbell Mcleay (Inactive)]

Oleg,

We have applied the provided patch but unfortunately that seems to be not helping much to reduce the evictions . Your thoughts on this please.