[LU-17303] mdt-aspls3-MDT0003 has many threads stuck in ldlm_completion_ast "client-side enqueue returned a blocked locksleeping"

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Labels: llnl
• Environment: server (asp4) lustre-2.14.0_21.llnl-5.t4.x86_64= ; clients (oslic) lustre-2.12.9_6.llnl-2.t4.x86_64, (ruby) lustre-2.12.9_7.llnl-1.t4.x86_64 ; TOSS 4.6-6
• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Gian-Carlo Defazio

Description

mdt-aspls3-MDT0003 is stuck and not responding to clients. It has many (~244) threads stuck in ldlm_completion_ast, stopping and starting lustre does not fix the problem.

Comments

[Gian-Carlo Defazio]

We haven't seen this issue since applying the suggested lnet tunables to our relic clusters.

[Cameron Harr]

Serguei, Thank you very much for helping us nail down these LNet tunings. It's something we've wanted to do for a long time but had a hard time finding straightforward documentation on how to do so.

[Cameron Harr]

Yes, we've been trying to move to 2.15 for over a year but have kept running into complex LNet issues with orelic and other routers. Talking internally this morning, we want to try 2.15 again in case these new tunings resolve the problems we were seeing.

[Serguei Smirnov]

If you find out you want to achieve performance improvements, you may want to consider upgrading to version supporting socklnd conns_per_peer and/or check if socklnd nscheds increase is helpful.

[Cameron Harr]

I'm not, though that would be interesting to see what difference there would be. My main purpose here was to clean up the Lustre fabric and connection issues, and things do look significantly cleaner since implementing those changes.

[Serguei Smirnov]

Cameron, are you planning to run any performance tests to compare "before" and "after"?

[Cameron Harr]

Since setting those values yesterday, at least one of the nodes (asp4) that was logging lots of errors, cleaned up immediately and has remained clean since.