[LU-7971] client fails to umount -f

• Status: Resolved
• Project: Lustre
• Type: Bug
• Priority: Critical
• Reporter: Cliff White (Inactive)
• Assignee: WC Triage
• Resolution: Cannot Reproduce
• Severity: 3

Description

Soak error:
Clients and servers are running soak build 20160324 - 2.8 plus http://review.whamcloud.com/#/c/18813/
https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&spaceKey=Releases#SoakTestingonLola-20160324

Stopped soak due to separate issue.
Remounted all servers.
Ran umount -f on all clients to clear state.
Lola-34 would not amount
dimes:

Lustre: 4595:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1459465123/real 1459465123]  req@ffff8803d37e29c0 x1529700237235520/t0(0) o38->soaked-MDT0001-mdc-ffff881032296000@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1459465178 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 4595:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Lustre: Evicted from MGS (at MGC192.168.1.108@o2ib10_0) after server handle changed from 0x3514dbeb8b19d96b to 0xc17b0433aaa7b860
Lustre: MGC192.168.1.108@o2ib10: Connection restored to MGC192.168.1.108@o2ib10_0 (at 192.168.1.108@o2ib10)
Lustre: soaked-MDT0001-mdc-ffff881032296000: Connection restored to 192.168.1.109@o2ib10 (at 192.168.1.109@o2ib10)
Lustre: 4595:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1459465774/real 1459465774]  req@ffff88043f0af980 x1529700237238268/t0(0) o400->soaked-MDT0000-mdc-ffff881032296000@192.168.1.108@o2ib10:12/10 lens 224/224 e 1 to 1 dl 1459465776 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1
Lustre: 4595:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 93 previous similar messages
Lustre: setting import soaked-MDT0000_UUID INACTIVE by administrator request
LustreError: 4595:0:(client.c:1143:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8801044fa0c0 x1529700237240872/t0(0) o38->soaked-MDT0000-mdc-ffff881032296000@192.168.1.108@o2ib10:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
LustreError: 220977:0:(file.c:3201:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
LustreError: 239794:0:(file.c:3201:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 239794:0:(file.c:3201:ll_inode_revalidate_fini()) Skipped 1 previous similar message
LustreError: 239791:0:(file.c:3201:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 239791:0:(file.c:3201:ll_inode_revalidate_fini()) Skipped 15 previous similar messages

Syslog:

Mar 31 16:19:07 lola-34 mrshd[239765]: [e-mail] as root: cmd='PATH=/scratch/apps/openmpi/1.8.1/bin/:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin;umount -f /mnt/soaked'
Mar 31 16:19:08 lola-34 kernel: Lustre: setting import soaked-MDT0000_UUID INACTIVE by administrator request
Mar 31 16:19:08 lola-34 kernel: LustreError: 4595:0:(client.c:1143:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8801044fa0c0 x1529700237240872/t0(0) o38->soaked-MDT0000-mdc-ffff881032296000@192.168.1.108@o2ib10:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
Mar 31 16:19:08 lola-34 kernel: LustreError: 220977:0:(file.c:3201:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -5
Mar 31 16:19:08 lola-34 mrshd[239764]: pam_unix(mrsh:session): session closed for user root
Mar 31 16:19:08 lola-34 xinetd[4025]: EXIT: mshell status=0 pid=239764 duration=1(sec)
Mar 31 16:19:08 lola-34 kernel: LustreError: 239794:0:(file.c:3201:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
Mar 31 16:19:08 lola-34 kernel: LustreError: 239794:0:(file.c:3201:ll_inode_revalidate_fini()) Skipped 1 previous similar message
Mar 31 16:19:09 lola-34 kernel: LustreError: 239791:0:(file.c:3201:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
Mar 31 16:19:09 lola-34 kernel: LustreError: 239791:0:(file.c:3201:ll_inode_revalidate_fini()) Skipped 15 previous similar messages