[LU-2965] MDS evicted OST after attempting recovery.

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Critical
• Affects Version/s: Lustre 2.4.0
• Labels: LB
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: James A Simmons

Comments

[Peter Jones]

Thanks James - that is good news!

[James A Simmons]

For our last test shot IR worked flawlessly. I think we can close this ticket. If we encounter the bug again this ticket can be reopened.

[Jodi Levi (Inactive)]

Reducing priority until more information is available to understand the issue.

[Li Wei (Inactive)]

From spare.log (why is it called "spare", by the way?):

Mar 8 21:06:01 widow-spare06 kernel: [ 7478.823574] Lustre: routed1-OST00eb-osc-ffff8804319c7000: Connection restored to routed1-OST00eb (at 10.36.227.92@o2ib)
Mar 8 21:06:01 widow-spare06 kernel: [ 7478.845592] Lustre: routed1-OST01ab-osc-ffff8804319c7000: Connection restored to routed1-OST01ab (at 10.36.227.92@o2ib)
Mar 8 21:12:33 widow-spare06 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Mar 8 21:12:33 widow-spare06 kernel: [ 0.000000] Initializing cgroup subsys cpuset
[...]
Mar 8 21:13:34 widow-spare06 kernel: [ 120.446824] ipmi device interface
[EOF]

The client completed recoveries at least with OST00eb and OST01ab, but restarted for some reason. OST00eb last heard from the client at around 21:08:53---two to three minutes after the recovery finished. If this was not something else, the timing was close to the 150s ping interval, suggesting the request heard was probably a ping. The next ping should happen 150s to 300s later, which correspond to 21:11:23 and 21:13:53. Could this eviction merely because the client was simply not remounted after the restart? Any log available after 21:13:34?

[Oleg Drokin]

I guess the assumption clients stopped pinging is because they were evicted for not pinging

[James A Simmons]

10.36.227.198 was our 2.4 non cray client. I attached the log (spare.log) here.

[Li Wei (Inactive)]

James,

According to the oss-test-shot.log, only one client was evicted by the OSS around 21:30: "61445347-9977-82cd-59dd-430903b6625f (at 10.36.227.198@o2ib)". It seems c4-0c0s5n0 was "1546@gni", not the o2ib client. Is the log on the o2ib client still available?

Also, how did you infer that some clients had stopped pinging the servers, please?

[James A Simmons]

Looking at it logs at 175 seconds after every client recovered some stopped pinging the servers.

[James A Simmons]

Nope. I didn't test suppress pings at that time. I started the suppress pings test 2 hours later than when all the recover problems happened.

[Oleg Drokin]

I just confirmed with James that the actual problem is "even though recovery completed just fine after a server failure, clients were evicted later for some reason".

Given the timing of 1354 seconds - that's exactly how much passed since recovery - appears the clients were not pinging?
Were you already playing with suppress pinging at the time, I wonder?