[LU-6529] Server side lock limits to avoid unnecessary memory exhaustion

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Labels: llnl
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Christopher Morrone (Inactive)

Description

As seen in tickets like LU-5727, we currently rely almost entirely on the good aggregate behavior of the lustre clients to avoid memory exhaustion on the MDS (and other servers, no doubt).

We require the servers, the MDS in particular, to instead limit ldlm lock usage to something reasonable to avoid OOM conditions on their own. It is not good design to leave the MDS's memory usage entirely up to the very careful administrative limiting of ldlm lock usage limits across all of the client nodes.

Consider that some sites have many thousands of clients across many clusters where such careful balancing and coordinated client limits may be difficult to achieve. Consider also WAN usages, where some clients might not ever reside at the same organization as the servers. Consider also bugs in the client, again like LU-5727.

See also the attached graph showing MDS memory usage. Clearly the ldlm lock usage grows without bound, and other parts of the kernel memory usage are put under undue pressure. 70+ GiB of ldlm lock usage is not terribly reasonable for our setup.

Some might argue that the SLV code needs to be fixed, and I have no argument against pursuing that work. That could certainly be worked in some other ticket.

But even if SLV is fixed, we still require enforcement of good memory usage on the server side. There will always be client bugs or misconfiguration on clients and the server OOMing is not a reasonable response to those issues.

I would propose a configurable hard limit on the number of locks (or space used by locks) on the server side.

I am open to other solutions, of course.

Issue Links

• is related to: LU-7266 Fix LDLM pool to make LRUR working properly
• is related to: LU-6390 lru_size on the OSC is not honored
• is related to: LU-6929 typo in cfs_hash_for_each_relax()
• is related to: LU-1520 client fails MDS connection and stack threads on another client
• is related to: LU-6775 Reduce memory footprint of ldlm_lock and ldlm_resource
• is related to: LU-11672 improving lru_max_age policy when lru resize is disabled
• is related to: LU-14221 Client hangs when using DoM with a fixed mdc lru_size
• is related to: LU-11509 LDLM: replace client lock LRU with improved cache algorithm
• is related to: LU-14858 kernfs tree to dump/traverse ldlm lock resources for debug
• is related to: LU-14859 cancel client DLM locks from the server
• is related to: LU-14517 Decrease default lru_max_age value
• is related to: LU-5727 MDS OOMs with 2.5.3 clients and lru_size != 0
• is related to: LU-8209 glimpse lock request does not engage ELC to drop unneeded locks
• is related to: LU-17428 reduce default value for lru_max_age to 300s

Comments

[D. Marc Stearman (Inactive)]

Thanks. I'll close the issue.

[Christopher Morrone (Inactive)]

Patches. Yes.

[D. Marc Stearman (Inactive)]

Chris, do we have this patch in our local release?

[Joseph Gmitter (Inactive)]

All patches have landed for 2.8.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/16123/
Subject: LU-6529 ldlm: improve proc interface of lock reclaim
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 33b55f223a42f20916bc417f7e5a21f68b59cd02

[Niu Yawei (Inactive)]

Another bug: The proc files accept negative values. Negative values should be rejected.

Well, it's same as other Lustre proc files which rely on some basic helper functions. I think it's worth a new ticket to fix this.

I am also disappointed that the patch passed review with so little in the way of function comments. Aren't function comments are landing requirement?

I'll try to add more comments in the next patch.

Under Lustre 2.5.4 + local patches, we seem to be hitting the high lock limit prematurely, at least as far as we can tell from the number of ldlm_locks active on the slab. Is there some other way to get an idea of what the lustre server thinks is the current lock count?

There is a counter for the ldlm lock, but it's not exported, maybe I'd export it via proc for debug purpose (together with the proc interface changes).

In current Lustre, you can roughly get the number by adding up the /proc/fs/lustre/ldlm/namespaces/$target/pool/granted for all the MDT/OST targets on server.

[Christopher Morrone (Inactive)]

Under Lustre 2.5.4 + local patches, we seem to be hitting the high lock limit prematurely, at least as far as we can tell from the number of ldlm_locks active on the slab. Is there some other way to get an idea of what the lustre server thinks is the current lock count?

[Christopher Morrone (Inactive)]

I am also disappointed that the patch passed review with so little in the way of function comments. Aren't function comments are landing requirement?

[Christopher Morrone (Inactive)]

Another bug: The proc files accept negative values. Negative values should be rejected.

[Gerrit Updater]

Niu Yawei ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16123
Subject: LU-6529 ldlm: improve proc interface of lock reclaim
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cd15ce613f958e14fd8c8a01a97cdd67cb17e249