https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1414 Lustre <p>We hit the following LBUG on the cient at our customer site.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 7 01:42:31 alps1-36 kernel: [5827658.348638] LustreError: 23918:0:(llite_lib.c:1382:ll_delete_inode()) ASSERTION(inode-&gt;i_data.nrpages == 0) failed: inode=139398948/2615229393(ffff8818d28e6d50) nrpages=1199638, see http://jira.whamcloud.com/browse/LU-118 May 7 01:42:31 alps1-36 kernel: [5827658.348688] LustreError: 23918:0:(llite_lib.c:1382:ll_delete_inode()) LBUG </pre> </div></div> <p>As far as I can see <a href="https://jira.whamcloud.com/browse/LU-118" title="clear_inode: BUG_ON(inode-&gt;i_data.nrpages)" class="issue-link" data-issue-key="LU-118"><del>LU-118</del></a>, this is fixed and the patches were landed in lustre-1.8.6, but we hit again.<br/> <a href="https://jira.whamcloud.com/browse/LU-267" title="Kernel race could cause LU-118, clear_inode: BUG_ON(inode-&gt;i_data.nrpages)" class="issue-link" data-issue-key="LU-267"><del>LU-267</del></a> seems to be related ticket of this issue. Is <a href="https://jira.whamcloud.com/browse/LU-118" title="clear_inode: BUG_ON(inode-&gt;i_data.nrpages)" class="issue-link" data-issue-key="LU-118"><del>LU-118</del></a> workaround patch? do we need to fix the kernel to completely solve this issue?</p> SLES11SP1 - Client<br/> CentOS5.6 - Servers LU-1414

ll_delete_inode() LBUG

Bug Minor Resolved Cannot Reproduce Jinshan Xiong Shuichi Ihara Wed, 16 May 2012 09:41:55 +0000 Mon, 29 May 2017 03:44:37 +0000 Mon, 29 May 2017 03:44:37 +0000 Lustre 1.8.6 0 7 <p>Jinshan</p> <p>Does this seem to be related to the issues previously handled under LU118?</p> <p>Peter</p> <p>This definitely is not <a href="https://jira.whamcloud.com/browse/LU-118" title="clear_inode: BUG_ON(inode-&gt;i_data.nrpages)" class="issue-link" data-issue-key="LU-118"><del>LU-118</del></a> where a race occurred so only 1 page is leaving after truncate. But here it had 1199638 pages.</p> <p>Can I have a backtrace and recent error messages in lustre log?</p> <p>client's /var/log/messages.</p> <p>will get /tmp/lustre-log.1336326151.23918</p> <p>We saw the same issue. My point of view and analyze below.<br/> 1) LASSERT at ll_delete_inode, after truncate we have non zero -&gt;nrpages for address_space.<br/> 2) File for this inode is already closed.<br/> 3) issue happened in the file per process access, so this is no race between the client threads. (form psfio.c)<br/> 4) nrpages have decreased between LASSERT and crash time(for example crash №3 have 381 and 0), so someone do truncate pages.<br/> 5) The page which exist at address space after truncate and LASSERT looks valid for lustre</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kmem -g 200000000001058 FLAGS: 200000000001058 PAGE-FLAG BIT VALUE PG_referenced 3 0000008 PG_uptodate 4 0000010 PG_lru 6 0000040 PG_private 12 0001000 PG_slob_free 12 0001000 struct ll_async_page { llap_magic = 98764321, llap_write_queued = 0, llap_defer_uptodate = 0, llap_origin = 3, llap_ra_used = 0, llap_ignore_quota = 0, llap_reserved = 0, llap_pglist_cpu = 0, llap_cookie = 0xffff880106be60e8, llap_page = 0xffffea0004f58908, llap_pending_write = { next = 0x0, prev = 0x0 }, llap_pglist_item = { next = 0xffff880106be7e28, prev = 0xffff880106be6218 }, llap_checksum = 0, llap_lockh_granted = { cookie = 0 } } </pre> </div></div> <p>No dirty, no lock, no writeback.<br/> lapp_origin = 3 LLAP_ORIGIN_COMMIT_WRITE - simple write.</p> <p>Root cause.</p> <p>The first thread with LASSERT is ldlm_bl_61 - trace</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 30477:0:(llite_lib.c:1384:ll_delete_inode()) ASSERTION(inode-&gt;i_data.nrpages == 0) failed: inode=326383344/3842751679(ffff88011f44c210) nrpages=2583694, see http://jira.whamcloud.com/browse/LU-118 LustreError: 30477:0:(llite_lib.c:1384:ll_delete_inode()) LBUG Pid: 30477, comm: ldlm_bl_61 Call Trace: [&lt;ffffffff810072e9&gt;] try_stack_unwind+0x149/0x190 [&lt;ffffffff81005ca0&gt;] dump_trace+0x90/0x300 [&lt;ffffffffa01349b2&gt;] libcfs_debug_dumpstack+0x52/0x80 [libcfs] [&lt;ffffffffa0134f21&gt;] lbug_with_loc+0x71/0xe0 [libcfs] [&lt;ffffffffa04c32ef&gt;] ll_delete_inode+0x14f/0x150 [lustre] [&lt;ffffffff8111ac23&gt;] generic_delete_inode+0xb3/0x180 [&lt;ffffffff8111ad4d&gt;] generic_drop_inode+0x5d/0x80 [&lt;ffffffff811197cd&gt;] iput+0x5d/0x70 [&lt;ffffffffa04e7d46&gt;] ll_mdc_blocking_ast+0x1b6/0x5a0 [lustre] [&lt;ffffffffa026758d&gt;] ldlm_cancel_callback+0x5d/0xe0 [ptlrpc] [&lt;ffffffffa027ef0c&gt;] ldlm_cli_cancel_local+0x6c/0x340 [ptlrpc] [&lt;ffffffffa0283b3e&gt;] ldlm_cli_cancel+0x5e/0x3f0 [ptlrpc] [&lt;ffffffffa04e7d87&gt;] ll_mdc_blocking_ast+0x1f7/0x5a0 [lustre] [&lt;ffffffffa028763c&gt;] ldlm_handle_bl_callback+0xbc/0x270 [ptlrpc] [&lt;ffffffffa0287981&gt;] ldlm_bl_thread_main+0x191/0x3e0 [ptlrpc] [&lt;ffffffff810035ba&gt;] child_rip+0xa/0x20 </pre> </div></div> <p>the second is ldlm_bl_67 - trace</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 31380 TASK: ffff88069c4c90c0 CPU: 28 COMMAND: "ldlm_bl_67" #0 [ffff8807f5055ad0] schedule at ffffffff812bb7f5 #1 [ffff8807f5055ba8] __cond_resched at ffffffff8103ea65 #2 [ffff8807f5055bc8] __cond_resched_lock at ffffffff8103eab7 #3 [ffff8807f5055be8] cache_remove_lock at ffffffffa03c9eae [osc] #4 [ffff8807f5055c48] osc_extent_blocking_cb at ffffffffa03b3606 [osc] #5 [ffff8807f5055cb8] ldlm_cancel_callback at ffffffffa026758d [ptlrpc] #6 [ffff8807f5055cd8] ldlm_cli_cancel_local at ffffffffa027ef0c [ptlrpc] #7 [ffff8807f5055d38] ldlm_cli_cancel at ffffffffa0283b3e [ptlrpc] #8 [ffff8807f5055db8] osc_extent_blocking_cb at ffffffffa03b34cf [osc] #9 [ffff8807f5055e28] ldlm_handle_bl_callback at ffffffffa028763c [ptlrpc] #10 [ffff8807f5055e78] ldlm_bl_thread_main at ffffffffa0287981 [ptlrpc] #11 [ffff8807f5055f48] kernel_thread at ffffffff810035ba </pre> </div></div> <p>From the stack of ldlm_bl_67 I got ldlm_lock pointer</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bt -f .... #1 [ffff8807f5055ba8] __cond_resched at ffffffff8103ea65 ffff8807f5055bb0: ffffffffffffff10 ffff88010456b568 &lt;-- this is spinlock_t l_extents_list_lock; ffff8807f5055bc0: ffff8807f5055be0 ffffffff8103eab7 #2 [ffff8807f5055bc8] __cond_resched_lock at ffffffff8103eab7 ffff8807f5055bd0: ffff88061718e640 ffff88061718e640 ffff8807f5055be0: ffff8807f5055c40 ffffffffa03c9eae #3 [ffff8807f5055be8] cache_remove_lock at ffffffffa03c9eae [osc] ffff8807f5055bf0: ffff880815d647c0 ffff88010456b400 ffff8807f5055c00: ffff880815d647d8 ffff88010456b570 ffff8807f5055c10: ffff88010456b568 ffff88011f44c210 .... crash&gt; p/x (0xffff88010456b568 -360) $18 = 0xffff88010456b400 crash&gt; ldlm_lock 0xffff88010456b400 struct ldlm_lock { l_handle = { h_link = { next = 0xffffc90010719050, prev = 0xffffc90010719050 }, h_cookie = 17652665684890613763, h_addref = 0xffffffffa0262210 &lt;lock_handle_addref&gt;, h_lock = { raw_lock = { slock = 1542 } }, ... l_lvb_len = 40, l_lvb_data = 0xffff88012d525e40, l_lvb_swabber = 0xffffffffa02b1c20, l_ast_data = 0xffff88011f44c210, &lt;--- this is inode l_extents_list_lock = { raw_lock = { slock = 6425 } }, ..... </pre> </div></div> <p>So we can see inode from ldlm_lock is the same as LASSERT inode ffff88011f44c210.<br/> Now, how race happend:<br/> 1) ll_mdc_blocking_ast()&gt;iput()&gt;...&gt;ll_delete_inode()&gt;truncate_inode_pages()<br/> 2) osc_extent_blocking_cb()&gt;...&gt;ll_page_removal_cb()&gt;ll_truncate_complete_page()&gt;truncate_complete_page()</p> <p>Close old ticket.</p> Development End date Thu, 15 Nov 2012 09:41:55 +0000 Rank 1|hzw14f: Rank (Obsolete) 10263 Severity Start date Wed, 16 May 2012 09:41:55 +0000