[LU-9372] OOM happens on OSS during Lustre recovery for more than 5000 clients

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.6
• Labels: cea
• Environment: Server running with b2_7_fe; Clients are a mix of IEEL3 (RH7/SCS5), 2.5.3.90 (RH6/AE4), 2.7.3 (CentOS7)
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Bruno Faccini (Inactive)

Description

I have been on-site to work with Bruno Travouillon (Atos) on one of the crash-dumps they have.

After joint analysis, it looks like a huge memory part is being consumed by "ptlrpc_request_buffer_desc" (17KB size each due to the embedded req, and that have been allocated in 32KB Slabs to increase/double side effect!).

Having a look to the concerned source code, it looks like these "ptlrpc_request_buffer_desc" could be additionally allocated upon need by ptlrpc_check_rqbd_pool(), but will never be freed until OST umount/stop by ptlrpc_service_purge_all().

This problem has caused several OSS failovers to fail due to OOM.

Issue Links

• is duplicated by: LU-1099 Lustre OSS OOMs repeatedly
• is related to: LU-10603 ptlrpc_lprocfs_req_buffers_max_fops unused
• is related to: LU-13600 limit number of RPCs in flight during recovery
• is related to: LU-10803 req_buffers_max and req_history_max setting problems
• is related to: LU-10826 Regression in LU-9372 on OPA enviroment and no recovery triggered