https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13594 Lustre <p>It would be useful to register an OOM callback in Lustre using <tt>register_oom_notifier()</tt> (and deregister at shutdown with <tt>unregister_oom_notifier()</tt>, firstly in libcfs and obdclass to print the current <tt>libcfs_kmemory</tt> and <tt>memused_show()</tt>/<tt>memused_max_show()</tt>, as well as potentially trying to shrink caches (e.g. the number of LNet message buffers, debug logs, etc.) before a userspace process is killed.</p> LU-13594

register OOM callback in Lustre

Improvement Minor Reopened Unresolved Etienne Aujames Andreas Dilger easy Fri, 22 May 2020 22:29:20 +0000 Wed, 22 Jun 2022 19:38:37 +0000 Lustre 2.14.0 0 3 <p>Arshad Hussain (arshad.hussain@aeoncomputing.com) uploaded a new patch: <a href="https://review.whamcloud.com/42121" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/42121</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13594" title="register OOM callback in Lustre" class="issue-link" data-issue-key="LU-13594">LU-13594</a> obdclass: Add OOM handler for obdclass<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2e61cf0c6c84608ea583ce342270746c84de7b69</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/42121/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/42121/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13594" title="register OOM callback in Lustre" class="issue-link" data-issue-key="LU-13594">LU-13594</a> obdclass: Add OOM handler for obdclass<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 54d4cca6cb0c92a09b364974438d91d4331a036f</p> <p>Landed for 2.15</p> <p>I've seen this callback a few times recently, running <a href="https://testing.whamcloud.com/search?horizon=2332800&amp;status%5B%5D=CRASH&amp;test_set_script_id=570ba67a-4a46-11e0-a7f6-52540025f9af&amp;sub_test_script_id=c8c35916-ad31-48df-b959-1a050638b897&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">sanityn test_56 on ZFS</a> but the current code just prints a brief message without much context and does nothing else:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[16506.409968] obd_memory max: 200336259, obd_memory current: 200336259 [16506.975974] obd_memory max: 200416739, obd_memory current: 200416739 [16507.013294] obd_memory max: 200416739, obd_memory current: 200416739 [16507.020553] obd_memory max: 200416739, obd_memory current: 200416739 [16507.035227] obd_memory max: 200416739, obd_memory current: 200416739 [16507.218562] obd_memory max: 200471595, obd_memory current: 200471595 [16507.224060] obd_memory max: 200471595, obd_memory current: 200471595 [16507.226494] obd_memory max: 200471595, obd_memory current: 200471595 [16507.229583] obd_memory max: 200471595, obd_memory current: 200471595 [16507.231476] obd_memory max: 200471595, obd_memory current: 200471595 </pre> </div></div> <p>It would be better if this message was prefixed with "{{Lustre: OOM handler: }}" to give some context to what it means.</p> <p>Secondly, having the handler itself at least provides some minimal information (Lustre memory usage is 200MB in this case, on a 3GB VM, not including LNet memory usage which should also be printed).</p> <p>It would be better if this callback actually tried to do something useful under memory pressure. Possible candidates would be:</p> <ul class="alternate" type="square"> <li>reduce number of server threads to free per-thread allocations</li> <li>cancel DLM locks on server (see <a href="https://jira.whamcloud.com/browse/LU-6529" title="Server side lock limits to avoid unnecessary memory exhaustion" class="issue-link" data-issue-key="LU-6529"><del>LU-6529</del></a> and related tickets)</li> <li>cancel DLM locks on client (drop LRU completely, if not already done)</li> <li>drop cached pages on client</li> </ul> Related LU-12830 LU-14456 LU-15963 Development Rank 1|i0114n: Rank (Obsolete) 9223372036854775807