https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10826 Lustre <p>Looks like <a href="https://jira.whamcloud.com/browse/LU-9372" title="OOM happens on OSS during Lustre recovery for more than 5000 clients" class="issue-link" data-issue-key="LU-9372"><del>LU-9372</del></a> has regression on OPA enviroment and somehow when test_req_buffer_pressure is enabled (test_req_buffer_pressure=1) on OSS or MDS, client never reconnect and no trigger Lustre recovery. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@es14k-vm1 ~]# umount -t lustre -a [root@es14k-vm1 ~]# lustre_rmmod [root@es14k-vm1 ~]# vi /etc/modprobe.d/lustre.conf options ptlrpc test_req_buffer_pressure=1 [root@es14k-vm1 ~]# mount -t lustre /dev/ddn/scratch0_ost0000 /lustre/scratch0/ost0000 [root@es14k-vm1 ~]# cat /proc/fs/lustre/obdfilter/scratch0-OST0000/recovery_status status: WAITING_FOR_CLIENTS </pre> </div></div> <p>Never client Reconnected and no lustre recovery triggered</p> master, centos7.4 <br/> 1 x MDS/MDT, 1 x OSS/OST and 1 client with OPA-10.6 LU-10826

Regression in LU-9372 on OPA enviroment and no recovery triggered

Bug Minor Resolved Fixed Bruno Faccini Shuichi Ihara Mon, 19 Mar 2018 14:45:33 +0000 Tue, 24 Jul 2018 17:50:11 +0000 Mon, 9 Apr 2018 21:00:43 +0000 Lustre 2.12.0 0 6 <p>Hmm, I may have been too restrictive in my <a href="https://jira.whamcloud.com/browse/LU-9372" title="OOM happens on OSS during Lustre recovery for more than 5000 clients" class="issue-link" data-issue-key="LU-9372"><del>LU-9372</del></a> patches when test_req_buffer_pressure is being configured.</p> <p>I will double-check on a test platform.</p> <p> </p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31690" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31690</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10826" title="Regression in LU-9372 on OPA enviroment and no recovery triggered" class="issue-link" data-issue-key="LU-10826"><del>LU-10826</del></a> ptlrpc: fix test_req_buffer_pressure behavior<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8420c6b4f04ee29c4374f9fdbe216c3f87e6fc26</p> <p>Thanks Bruno, patch <a href="https://review.whamcloud.com/31690" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31690</a> works and now triggred Lustre recovery.<br/> However, main concerns are tunning of resources allocation for all clients requests with test_req_buffer_pressure=1.<br/> In fact, as <a href="https://jira.whamcloud.com/browse/LU-9372" title="OOM happens on OSS during Lustre recovery for more than 5000 clients" class="issue-link" data-issue-key="LU-9372"><del>LU-9372</del></a> described, if single OSS has many OSTs under limited amount of memory resources, Lustre recovery triggers OOM when Lustre recovery started in parallel for many OSTs. We want to prevent OOM on that situation. However, curently, once we set test_req_buffer_pressure=1, it's controled by memory size or we use req_buffers_max for manuall setting. Still not not automanted setting.</p> <p>Shuichi, I am already looking for a way to auto-tune req_buffers_max based, possibly based on current number of active OSTs/targets, number of connected/to-be-recovered Clients, memory available, ...</p> <p>I also want to check some others options which may help to limit each rqbd-buffer memory footprint (presently allocated in a 32K slab when it size is only 17K (this since patch for <a href="https://jira.whamcloud.com/browse/LU-4755" title="ASSERTION( req-&gt;rq_reqbuf_len &gt;= msgsize ) failed when using 4MB RPC" class="issue-link" data-issue-key="LU-4755"><del>LU-4755</del></a> "ptlrpc: enlarge OST_MAXREQSIZE for 4MB RPC"), like to use a dedicated kmem_cache (if no underlying interference/drift from the Slab/Slub layer), or try to reduce OST_MAXREQSIZE in order to have a full buffer size down to 16K (if it keeps the "4MB RPC" capability) and thus have it being allocated in a 16K Slab.</p> <p> </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31690/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31690/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10826" title="Regression in LU-9372 on OPA enviroment and no recovery triggered" class="issue-link" data-issue-key="LU-10826"><del>LU-10826</del></a> ptlrpc: fix test_req_buffer_pressure behavior<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 040eca67f8d5422b0099d1b70594b5eb40a0f9ef</p> <p><font color="#000000">Landed for 2.12</font></p> Related LU-9372 LU-10993 Development Rank 1|hzzujb: Rank (Obsolete) 9223372036854775807 Severity