https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6587 Lustre <p>Since vmalloc allocations are notoriously unscalable and slow, we should do what other filesystems are doing (and what we converted upstream kernel client to do) and for large allocations we should always try kmalloc first (with NOWARN flag) and then if that fails, retry with vmalloc.</p> LU-6587

refactor OBD_ALLOC_LARGE to always do kmalloc first

Bug Minor Resolved Fixed Yang Sheng Oleg Drokin Mon, 11 May 2015 12:07:35 +0000 Fri, 5 Feb 2016 15:35:17 +0000 Fri, 5 Feb 2016 15:35:17 +0000 Lustre 2.8.0 0 14 <p>Yang Sheng is creating the patch and will push it to Gerrit.</p> <p>Yang Sheng (yang.sheng@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14994" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14994</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6587" title="refactor OBD_ALLOC_LARGE to always do kmalloc first" class="issue-link" data-issue-key="LU-6587"><del>LU-6587</del></a> obd: refactor OBD_ALLOC_LARGE marco<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a72771fc367cb430626bb6bf885fabb7a4cd945e</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14994/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14994/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6587" title="refactor OBD_ALLOC_LARGE to always do kmalloc first" class="issue-link" data-issue-key="LU-6587"><del>LU-6587</del></a> mem: refactor OBD_ALLOC_LARGE marco<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 919b85d796f845ec3e6633702dda1639a3d86176</p> <p>The next step of this ticket is replacing OBD_FREE_LARGE to OBD_FREE. Either all in one or one by one as subsystem. But i think we need a grace period for every one know that. So could we close this ticket first?</p> <p>Looking at patches by their sublayer the areas under going the most changes are llite, osp, and target. I would recommend doing a patch set with that combination that could perhaps be landed after feature freeze. The rest of the OBD_FREE_LARGE users appear to be safe to update. I noticed ptlrpc and obdclass are large users of OBD_FREE_LARGE so if you might wanted to do a patch set covering those regions.</p> <p>Thanks for your comment, James. I think it is good idea to make a patch against stable code first.</p> <p>It looks like this is landed for 2.8</p> <p>I noticed that there was something landed in the <a href="http://review.whamcloud.com/14994" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14994</a> patch that I don't really like. Moving <tt>is_vmalloc_addr()</tt> into OBD_FREE() means that there is the overhead of this check for every call to OBD_FREE(), even though this is not needed for the large majority of allocations. It would be better to have the <tt>is_vmalloc_addr()</tt> check only in OBD_FREE_LARGE() and remove it from OBD_FREE(). The deprecation of OBD_FREE_LARGE() should also be removed from checkpatch.pl.</p> <p>There are a couple of patches that have landed since then that need to be converted back to using OBD_FREE_LARGE(), at least:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>af13cfff297d4882de35fb7c11bf5261293b8287 09141c0796802e7a3471c084ea5928674b3a1862 </pre> </div></div> <p>Since this is only a few lines of change, it would be good to get this fixed before 2.9 if possible.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18034" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18034</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6587" title="refactor OBD_ALLOC_LARGE to always do kmalloc first" class="issue-link" data-issue-key="LU-6587"><del>LU-6587</del></a> obdclass: use OBD_FREE_LARGE with OBD_ALLOC_LARGE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a7ac09582c72dad0be99db2072971dd34d6d6165</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> inline <span class="code-object">int</span> is_vmalloc_addr(<span class="code-keyword">const</span> void *x) { #ifdef CONFIG_MMU unsigned <span class="code-object">long</span> addr = (unsigned <span class="code-object">long</span>)x; <span class="code-keyword">return</span> addr &gt;= VMALLOC_START &amp;&amp; addr &lt; VMALLOC_END; #<span class="code-keyword">else</span> <span class="code-keyword">return</span> 0; #endif } </pre> </div></div> <p>So I believe there's next to no overhead in is_vmalloc_addr esp. compared to other stuff we do in OBD_FREE()</p> <p>I'd be instead inclined to drop OBD_ALLOC_LARGE and just always do OBD_ALLOC that would fallbacl to vmalloc if regular alloc failed?</p> <p>Sure there is little overhead to check <tt>is_vmalloc_addr()</tt> for every single allocation, but not zero, so why would we use it everywhere when it is not needed. Also, in the upstream kernel these macros are gone and we don't want to have the <tt>vmalloc((</tt> fallback and calls to <tt>is_vmalloc_addr()</tt> check <em>everywhere</em> in the code simply because we don't know anymore which allocations might be large and which ones are not. We'd be taking a huge step backwards in terms of code quality, and a patch like that would never be accepted upstream, so why would we do it in master just because it can hide inside a macro?</p> <p>Having huge allocations that require vmalloc() is something we should be very aware of and make a conscious decision to do, as otherwise we might be wasting a lot of memory accidentally.</p> <p>Doesn't OBD_ALLOC_LARGE always try a kmalloc first then if it fails try a vmalloc? I don't think we can just drop the is_vmalloc_addr so easily. I agree with you Andreas that the vmalloc fallbacks are bad news but instead of trying to modify OBD_FREE_LARGE why don't we ripe off the band aid and start using vmalloc directly when it really needed. Note the upstream client has function wrapper libcfs_kvzalloc() and libcfs_kvzalloc_cpt() in libcfs to replace all those macros. Honestly I like to just use vmalloc directly.</p> <p>James: The problem with vmalloc is that it is expensive, it's all done under a single lock for one.<br/> As such trying kmalloc first has a big advantage of speed should there be big enough chunk of memory really available.</p> <p>Andreas: You are right that the overhead is there, even though it is pretty much negligible and for as long as we retain OBD_ALLOC_LARGE/FREE it makes the most sense to have the check in there and not in the OBD_ALLOC/FREE</p> <p>James, I'd be happy to use the libcfs_kvzalloc() and libcfs_kbfree() in master as well. We definitely can't use vmalloc() for everything in Lustre, since the performance penalty would be very high as Oleg wrote. Those helpers should still be wrapped as OBD_ALLOC_LARGE() and OBD_FREE_LARGE() to account memory allocations so that we can find memory leaks in the code. In either case, we still need to know that "large" allocations need to be freed differently from normal ones. </p> <p>Sorry I wasn't completely clear at what I was purposing. I was considering the idea of dropping vmalloc usage completely unless the allocation is more than 4MB. It was only those cases I would consider looking at vmalloc usage directly. Sorry for the confusion.</p> <p>As for memory accounting that is a separate topic for 2.9 but it is a good topic. We really should look at more standard ways to detect kernel leaks which are far more powerful than what lustre does today. I will open a separate ticket for that.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18034/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18034/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6587" title="refactor OBD_ALLOC_LARGE to always do kmalloc first" class="issue-link" data-issue-key="LU-6587"><del>LU-6587</del></a> obdclass: use OBD_FREE_LARGE with OBD_ALLOC_LARGE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0b1ad400c8f64575292a7ff54a8ce872a124b19e</p> <p>Patch has landed for 2.8</p> Related LU-7666 Development Rank 1|hzxcxb: Rank (Obsolete) 9223372036854775807 Severity