https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2084 Lustre <p>While working with router buffers, I set the number of large buffers to a number beyond the amount of memory I had assigned to the VM running Lustre. Number of large buffer: 1024, amount of memory: 1G. The VM froze with all 3 virtual cpu's running at 100%.</p> <p>Looking deeper into this, I found that the Linux memory allocation system will keep trying to free up memory to satisfy the request. However, even after waiting 15 minutes, the VM did not "unfreeze".</p> <p>I changed the default flags we use for memory allocation to include __GFP_NORETRY to stop the memory allocator from looping. When re-running the above test, I found the system no longer froze but returned -ENOMEM to the caller as expected.</p> <p>This bug is to track a discussion as to whether we should start using __GFP_NORETRY and if so, how widespread.</p> LU-2084

Kernel freeze allocating more memory than there is RAM

Bug Major Resolved Fixed Andreas Dilger Doug Oucharek Wed, 3 Oct 2012 19:20:24 +0000 Wed, 27 Oct 2021 03:48:00 +0000 Wed, 27 Oct 2021 03:48:00 +0000 Lustre 2.2.0 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.1.3 Lustre 1.8.8 Lustre 2.15.0 0 7 <p>In general:</p> <p>I have seen near OOM situations take hours (think overnight to days) to work themselves out if it is possible, over committal of kernel size code fairs badly. 15min is just getting started with these things. In your "frozen" state your kernel was likely not broken in any way just busy trying to accomplish what you asked it to do. </p> <p>There are going to be some critical sections of code that cannot fail and should block until fulfilled. It is definitely a per-allocation-class question of weather it should block or not. If you are allocating a whole systems amount of memory it should use __GFP_NORETRY and check for -ENOMEM, also I would hope that resources users this large would carefully grow as needed.</p> <p>Where did you add the __GFP_NORETRY flag? </p> <p>Given how there are many layers of "abstraction" in libcfs for OS portability, I ended up just hardcoding the addition of __GFP_NORETRY in routine cfs_alloc_flags_to_gfp() in linux-mem.c.</p> <p>Since we don't preallocate everything and do have some level of dynamic allocation, the possibility will exist that a happy running server could all of a sudden appear to freeze. I myself thought I had crashed the kernel since the terminal was no longer responsive. Only by seeing the CPU meter on the host system did I notice that the VM is still running at 100%.</p> <p>From a user's perspective, I would rather have an error message saying "task X could not be done because of a lack of memory" rather than a freeze. </p> <p>Yes working to keep the system out of OOM is a much better user experience.</p> <p>cfs_alloc_flags_to_gfp seems to be pretty low. I would think that is a huge amount of code affected. <br/> Are you using cfs_alloc(size_t nr_bytes, u_int32_t flags)? You can passdown __GFP_NORETRY to your specific allocation. </p> <p>What are you seeing as your -ENOMEM indication? </p> <p>Doug, wouldn't it make sense to limit the number of router buffers to some amount less than the total amount of RAM? Using __GFP_NORETRY in a blanket fashion seems like it could cause gratuitous system failures for cases where there is low memory, but the allocation is not absurd like in your case.</p> <p>1. I think __GFP_NORETRY is reasonable for router buffers. Routers should be dedicated nodes, where there's nothing else running - i.e. there's nothing like dirty pages to be flushed or idle process pages to be swapped out, so it makes little sense to make the VM retry.</p> <p>2. I don't think we should make it fool proof by limiting large_router_buffers. System administrators should understand what large_router_buffers does, if they ask for too much, they are asking for trouble and should end up with troubles. Such failures happen only once at router startup, and such routers would be avoided by clients and servers by router pingers, so the consequence should not be catastrophic. Then the admin should notice it and learn his lesson.</p> <p>This becomes more complicated when looking forward to the Dynamic LNet Config project which will be making the router buffer pools changeable. With the code as is today, if a user tells a running router to increase the size of a pool beyond available memory, we will see the router lock up for potentially hours. That is unacceptable.</p> <p>If we use __GFP_NORETRY, it may return ENOMEM in cases where memory could have been freed to satisfy the request. However, I would rather see this than a live router lockup.</p> <p>Checking ahead of time to see if there is RAM available does not sound easy given how the Linux memory manager works. Also, I feel this would be doing the OS's job for it.</p> <p>I heard somewhere that work was done to the memory manager in the Linux 3 streams to address these sort of issues. None of that was back-ported to 2.6. </p> <p>I tend to think __GFP_NORETRY is sufficient. On dedicated routers, where could the VM free much memory from?</p> <p>Good point.</p> <p>Ok, I can add CFS_ALLOC_NORETRY to our own set of memory allocation flags and map this to __GFP_NORETRY when present. This way it can be added on a case by case basis. I will only add this flag when allocating router buffers.</p> <p>As much as we could wish everyone using Lustre understood it as well as the developers, I don't think this is at all realistic. Users need to be told that something they are trying to do is unrealistic, rather than causing failures or hanging/crashing the node. Having a check like the following seems reasonable:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (router_buffer_pages &gt; cfs_num_physpages * 7 / 8) { CERROR(<span class="code-quote">"too much router memory requested: max %u\n"</span>, cfs_num_physpages * 7 / 8); RETURN(-EINVAL); fi </pre> </div></div> <p>with allowances for printing messages with proper units, etc. We still need to keep some memory for other things as well, which we may not get with a simple -ENOMEM case.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45174" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45174</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2084" title="Kernel freeze allocating more memory than there is RAM" class="issue-link" data-issue-key="LU-2084"><del>LU-2084</del></a> lnet: don't retry allocating router buffers<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ebd97d4585eea1aa7717f555a52dc24bcfa1885e</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45174/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45174/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2084" title="Kernel freeze allocating more memory than there is RAM" class="issue-link" data-issue-key="LU-2084"><del>LU-2084</del></a> lnet: don't retry allocating router buffers<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3038917f12a53b059473db172f5126136e20abc0</p> <p>Landed for 2.15</p> Development Rank 1|hzv51j: Rank (Obsolete) 4350 Severity