[LU-10961] Clients hang after failovers. LustreError: 223668:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -4

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.12.0
• Affects Version/s: Lustre 2.12.0
• Labels: soak
• Environment: soak cluster
• Severity: 3
• Assignee: Mikhail Pershin
• Reporter: Cliff White (Inactive)

Description

We are seeing repeated hard hang on clients after server failover.
'df' on a client will hang, user tasks do no complete. So far no hard faults, the node just grinds to a halt. Yesterday this occurred on soak-17 and soak-23. I have dumped stacks on both nodes, and crash dumps are available on soak.
We see:

• connections to one or more osts drop, and the client does not re-connect:

  Apr 27 03:28:42 soak-23 kernel: Lustre: 2084:0:(client.c:2099:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1524799714/real 0]  req@ffff8808fee67500 x1598738343197024/t0(0) o400->soaked-OST000b-osc-ffff8807f6ba0800@192.168.1.107@o2ib:28/4 lens 224/224 e 0 to 1 dl 1524799721 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
  Apr 27 03:28:42 soak-23 kernel: Lustre: soaked-OST0011-osc-ffff8807f6ba0800: Connection to soaked-OST0011 (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete
  Apr 27 03:28:42 soak-23 kernel: Lustre: Skipped 3 previous similar messages
  Apr 27 03:28:42 soak-23 kernel: Lustre: 2084:0:(client.c:2099:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
  Apr 27 03:28:42 soak-23 kernel: Lustre: soaked-OST000b-osc-ffff8807f6ba0800: Connection to soaked-OST000b (at 192.168.1.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete
  

  As of 1700 hours (14 hours after failover) the node still has not reconnected to this OST.

We also see repeated errors referencing the MDT:

Apr 27 17:25:42 soak-23 kernel: LustreError: 223668:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -4

The error appears very repeatable. Logs and stack traces are attached.

Issue Links

• is blocked by: LU-11158 PFL component instantiation is not replayed properly

Comments

[Peter Jones]

Landed for 2.12

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32791/
Subject: LU-10961 ldlm: don't cancel DoM locks before replay
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b44b1ff8c7fc50d5b0438926c89c69eb85817168

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32889
Subject: LU-10961 ldlm: extended testing
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 43febdf398958c18590fc6a9d306a615b907305b

[Mikhail Pershin]

It seems that problem is not in DOM but PFL. When a new component is instantiated during failover then that is not replayed properly. It is worth to create new ticket for that.
That explains also why Cliff reported problems with PFL-only setup as well.

[Mikhail Pershin]

There is another problem with replay - if DOM write cause new component instantiation then replay of both layout change and write RPC failed. I am investigating that.

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32791
Subject: LU-10961 ldlm: don't cancel DoM locks before replay
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5c5ef9be5904a7f16aa41a341c561b0c9c6beb42

[Mikhail Pershin]

I was able to reproduce lock replay issue with DoM files, patch will be ready soon. The problem is the lock cancellation before lock replay, MDC cancels locks differently than OSC and that causes DoM locks to be handled improperly.

Meanwhile that doesn't explain failover failure with PFL-only file because there is no MDC involved, so there can be another reasons for that. 

[Mikhail Pershin]

Peter,

yes, there are clues, I am working on reproducer and patch

[Peter Jones]

Mike

Has the information provided any insight?

Peter

[Cliff White (Inactive)]

Console logs from soak-17,42 and MDS (soak-8) attached.