https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4194 Lustre <p>We have a lustre client dedicated to the task of running robinhood. This node is running lustre <a href="https://github.com/chaos/lustre/tree/2.4.0-18chaos" class="external-link" target="_blank" rel="nofollow noopener">2.4.0-18chaos</a> and talking to our 55PB zfs-osd filesystem running a similar tag of lustre.</p> <p>The lustre client node that runs robinhood is hitting a "BUG: unable to handle kernel paging request at &lt;pointer&gt;" as a result of attempting to kfree() a bad pointer in lustre's ldlm_lock_put() function.</p> <p>There are two stacks that we have seen lead up to this, both ending in the same failed kfree.</p> <p>The first is from ldlm_bl_* threads:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cfs_free ldlm_lock_put ldlm_cli_cancel_list ldlm_bl_thread_main </pre> </div></div> <p>The second was under a robinhood process:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cfs_free ldlm_lock_put ldlm_cli_cancel_list ldlm_prep_elc_req ldlm_prep_enqueue_req mdc_intent_getattr_pack mdc_enqueue mdc_intent_lock lmv_intent_lookup lmv_intent_lock ll_lookup_it ll_lookup_nd do_lookup __link_path_walk path_walk do_path_lookup user_path_at vfs_fstatat vfs_lstat sys_newlstat system_call_fastpath </pre> </div></div> <p>Note that it was the second call to ldlm_cli_cancel_list() in ldlm_prep_elc_req() that we were under.</p> <p>The problem is pretty reproducible on our system. The client node usually crashes in less than an hour. But I am not aware of how to reproduce this elsewhere. We have other server clusters with robinhood instances on clients that are not crashing.</p> Lustre [2.4.0-18chaos|<a href="https://github.com/chaos/lustre/tree/2.4.0-18chaos">https://github.com/chaos/lustre/tree/2.4.0-18chaos</a>] on the client. Similar on the servers, using ZFS OSDs. LU-4194

kfree fails kernel paging request under ldlm_lock_put()

Bug Minor Resolved Fixed Bruno Faccini Christopher Morrone mn4 Thu, 31 Oct 2013 22:16:41 +0000 Fri, 7 Feb 2014 08:52:46 +0000 Mon, 2 Dec 2013 17:07:57 +0000 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.5.1 0 8 <p>Bruno</p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hello Chris,<br/> Thanks to detail the context where this crash occurs. But more infos are required for sure.</p> <p>Since it is a crash you don't have the lustre_log available (BTW, do you know the debug_mask in place at the time of crash?), but do you know how to extract it from the crash-dump if any was taken ? Or can the dump be available ?</p> <p>By experience, I don't think problem comes from RobinHood itself but more due to its induced work-load. </p> <p>Also, do you use the "Lustre-aware" version of RH (ie, the one polling the Change-log) ?</p> <p>I can get the lustre log from the crash dump. We have an extension to crash that enables that. Unfortunately, I cannot share that log of the crash dump. I will continue to investigate.</p> <p>Our installation of robinhood is, of course, employing Lustre changelogs.</p> <p>According to the different threads/stacks you reported to trigger the same BUG()/Oops seems that we may face a race during ELC (aka Early Lock Cancel) process. And again this is likely to be triggered due to RobinHood work-load.</p> <p>When I continue to investigate this way, you may try to run with ELC feature disabled (/proc/fs/lustre/ldlm/namespaces/*/early_lock_cancel = 0) for sometimes and see if this work-around ?</p> <p>Also, have you been able to find something of interest in lustre-log extracted from crash-dump ?</p> <p>From the lustre log I got the pointer to the lock in question (I think). The l_lvb_* fields are interesting. They are something like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>l_lvb_type = LVB_T_NONE l_lvb_len = 18464 l_lvb_data = 0xffffc90046218000 </pre> </div></div> <p>Is it valid for there to be data there when the type is LVB_T_NONE?</p> <p>FYI, we still hit a kfree() failure with early_lock_cancel disabled. Although it was a new call path that triggered it this time. It failed under the ll_imp_inval thread something like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cfs_free ldlm_lock_put cleanup_resource ldlm_resource_cleanup cfs_hash_for_each_relax cfs_hash_for_each_nolock ldlm_namespace_cleanup mdc_import_event ptlrpc_invalidate_import ptlrpc_invalidate_import_thread </pre> </div></div> <p>So Chris, if I understand you correctly, the kfree() failure is not due to freeing the lock itself but more likely during invalid LVB freeing ?</p> <p>Have you been able to double-check it for the 2nd crash you indicated ?</p> <p>If it is, I agree, likely to be the case, the failing address (CR2 register in exception context of the Oops'ing thread stack) should be around l_lvb_data pointer value.</p> <p>I will review the locations in the code where l_lvb_data is allocated/freed/initialized and see where this (l_lvb_data still referenced/non-null but freed ?) can happen.</p> <p>Chris, just to be sure I work on the right source code, to get it, I did :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>git clone https://github.com/chaos/lustre.git git checkout 2.4.0-18chaos </pre> </div></div> <p>ist it ok ? I am asking because actually I don't find a path in source code that can lead to this situation.</p> <p>Also the size of 18464 bytes seems not usual for LVB … And LVB_T_NONE value is 0, so how were the others fields of the suspected ldlm_lock ? And did you find the same kind/range of values/pointers for the second crash-dump ?</p> <p>What is the answer of "crash" tool "kmem" sub-command when using the 0xffffc90046218000 pointer as the argument ?</p> <p>You indicate that BUG()/Oops occurs in kfree(), but is this what you learned from the error msgs in Console/crash-dump?</p> <p>Last, was there any msg of interest in the log/Console sometime before any crash ??</p> <p>I am sorry to ask you so many questions, but I try to consider all the possible scenarios.</p> <p>Humm, 18464 bytes could be something like a layout/LOV-EA for a stripe count of 768, does this seems reasonable with the config you use ?</p> <p>May be (got inputs from Johann and Jinshan about this possibility) the fact that there is LVB data with a type of LVB_T_NONE comes from layout_fetch(), but anyway this does not explain why the LVB reference can be wrong and this should come from some race …</p> <blockquote><p>So Chris, if I understand you correctly, the kfree() failure is not due to freeing the lock itself but more likely during invalid LVB freeing ?</p></blockquote> <p>Yes, I think it is failing here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> OBD_FREE(lock-&gt;l_lvb_data, lock-&gt;l_lvb_len);</pre> </div></div> <blockquote><p>Chris, just to be sure I work on the right source code, to get it, I did </p></blockquote> <p>Yes, that tag from github is correct. I'm using -19chaos now, but that is just because I used the top of the tree in testing, not because I thought the minor differences between -18chaos and -19chaos would make a difference for this bug.</p> <blockquote><p>Humm, 18464 bytes could be something like a layout/LOV-EA for a stripe count of 768, does this seems reasonable with the config you use ?</p></blockquote> <p>Yes, our OST (and OSS) count is exactly 768.</p> <p>Thanks for the answers/confirmations.</p> <p>My comment before about "the fact that there is LVB data with a type of LVB_T_NONE comes from layout_fetch()" is wrong and must be inverted, it can NOT come from layout_fetch() !!… but I think now can better come from un-expected/requested Layout embedding within LVB reply from Server, and this can happen at least in ldlm_handle_cp_callback().</p> <p>I am currently testing a fix for that.</p> <p>I'm glad that you have a theory, because I'm lost.</p> <p>In one of the recent tests, the kernel complains about a page fault at</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ffffeae380f57540</pre> </div></div> <p>The lustre log extracted from that crash dump tells me the the last message from the thread that hit the page fault Oops was doing the final ldlm_lock_put() on the ldlm_lock at this address:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ffff885e3d09ce00</pre> </div></div> <p>The lock-&gt;l_lvb_data pointer is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ffffc90046218000</pre> </div></div> <p>When I use crash's "rd" command to dump the contents of the l_lvb_data buffer, it is indeed poisoned with OBD_FREE's repeated bytes of 0x5a. The pointer to the buffer has not, on the other hand, been poisoned yet.</p> <p>Which all probably just confirms that we are in kfree() under cfs_free() under ldlm_lock_put() at the time of the Oops.</p> <p>But where does the page fault address of ffffeae380f57540 come from? I find it in both the RAX and CR2 registers under cfs_free from crash's backtrace output. RAX is a common register, and CR2 is the Page Fault Linear Address register, which makes sense because we Oops under the failed page fault.</p> <p>I just wish that I understood where kfree() was getting that particular address.</p> <p>It looks like the backtrace below kfree() where we hit the page fault failure looks a little like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>constant_test_bit test_bit virt_to_page virt_to_head_page virt_to_cache kfree </pre> </div></div> <p>At least crash claims that we are on the pointer dereference under constant_test_bit(), and looking forward and backwards a bit from kfree+0x173 seems to place us under virt_to_page().</p> <p>I started to try to calculate virt_to_page by hand, but that way lies madness. I am just going to assume that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>virt_to_page(0xffffc90046218000) = 0xffffeae380f57540</pre> </div></div> <p>All that doesn't really change anything. But I feel somewhat enlightened. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Hello Chris, thanks again for this low-level VM debugging !</p> <p>I am not 100% sure but according to your last inputs, I think there is something contradictory here, how crash tool allows you to read poisoned memory meaning it was able to do virt2phys translation, when the corresponding page-table entry address resolution (to find corresponding/owning kmem-cache), itself using the same virt2phys translation to compute the page-frame number and find offset in page-table, fails ???</p> <p>Again, can you tell me what returns the "kmem 0xffffc90046218000" sub-command ? Does-it return a page-table entry address of 0xffffeae380f57540 ??</p> <p>And patch attempt to set l_lvb_type coherent is at <a href="http://review.whamcloud.com/#/c/8270/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8270/</a>.</p> <blockquote><p>Again, can you tell me what returns the "kmem 0xffffc90046218000" sub-command ? Does-it return a page-table entry address of 0xffffeae380f57540 ??</p></blockquote> <p>No, it does not. But I don't understand what I am seeing yet. I'll transcribe it here and maybe you can help me:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; kmem 0xffffc90046218000 VM_STRUCT ADDRESS RANGE SIZE ffff885e3df9ed80 ffffc90046218000 - fffc9004621e000 24576 PAGE PHYSICAL MAPPING INDEX CNT FLAGS ffffea0149d59378 5e3d059000 0 0 1 c0000000000000 </pre> </div></div> <p>Brian Behlendorf provided me with a key insight:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#define VMALLOC_START _AC(0xffffc90000000000, UL) #define VMALLOC_END _AC(0xffffe8ffffffffff, UL) </pre> </div></div> <p>So 0xffffc90046218000, the address that we are attempting to kfree, is in the vmalloc address range.</p> <p>And sure enough, it looks like ldlm_lock_put() is using OBD_FREE(), which is kfree-only. However, there are a couple of functions, at least mdc_finish_enqueue() and ll_layout_fetch() that are allocating buffers using OBD_FREE_LARGE() and pointing to them from the ldlm_lock's l_lvb_data pointer.</p> <p>I am testing the following patch in which I attempt to use the alloc and free functions a bit more consistently.</p> <p><a href="http://review.whamcloud.com/8298" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8298</a></p> <p>I am not entirely sure that we want to use OBD_ALLOC_LARGE all the time in ldlm_lock_create(). I don't know how often the buffers will be large enough to switch to vmalloc. The fix could certainly be implemented other ways. But this should suffice for testing.</p> <p>This code really needs some cleanup work.</p> <p>Robinhood has been running for two hours one the client with patch <a href="http://review.whamcloud.com/8298" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8298</a> without a problem. That looks like our problem.</p> <p>I will be away at SC'13 all of next week. If you could take over working on a finalized version of <a href="http://review.whamcloud.com/8298" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8298</a> while I am gone, that would be great.</p> <p>Great job!! BTW, the fact it was an address from the VMALLOC range becomes obvious with the kmem sub-command translation.</p> <p>Landed for 2.6</p> <p>It looks like we just hit this bug with one of our 2.4 clients.</p> <p>Patches <a href="http://review.whamcloud.com/8270" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8270</a> and <a href="http://review.whamcloud.com/8298" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8298</a> were cherry-picked to Lustre b2_5 branch.</p> Development Rank 1|hzw7i7: Rank (Obsolete) 11361 Severity