https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11582 Lustre <p>Hi,<br/> We have a type of job that keeps crashing Lustre client version 2.10.5 with the following trace. It is very likely this job has files open on Oak (Lustre 2.10.4). This looks like old tickets <a href="https://jira.whamcloud.com/browse/LU-1414" title="ll_delete_inode() LBUG" class="issue-link" data-issue-key="LU-1414"><del>LU-1414</del></a> and <a href="https://jira.whamcloud.com/browse/LU-118" title="clear_inode: BUG_ON(inode-&gt;i_data.nrpages)" class="issue-link" data-issue-key="LU-118"><del>LU-118</del></a>... fixed in Lustre 1.8! The issue happens on a bigmem node (1.5TB of RAM), and doesn't seem to happen on node with less memory. I'll try to upload a crash dump file to your ftp.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[11497.465606] LustreError: 132407:0:(llite_lib.c:2047:ll_delete_inode()) ASSERTION( inode-&gt;i_data.nrpages == 0 ) failed: inode=[0x200018e83:0x1ba2c:0x0](ffff8aa85a298510) nrpages=1, see LU-118 [11497.487939] LustreError: 132407:0:(llite_lib.c:2047:ll_delete_inode()) LBUG [11497.495730] Pid: 132407, comm: spades 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 [11497.505939] Call Trace: [11497.508685] [&lt;ffffffffc09947cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [11497.516009] [&lt;ffffffffc099487c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [11497.522955] [&lt;ffffffffc0f25c87&gt;] ll_delete_inode+0x1b7/0x1c0 [lustre] [11497.530291] [&lt;ffffffff8d43c504&gt;] evict+0xb4/0x180 [11497.535663] [&lt;ffffffff8d43ce0c&gt;] iput+0xfc/0x190 [11497.540940] [&lt;ffffffff8d43126e&gt;] do_unlinkat+0x1ae/0x2d0 [11497.546990] [&lt;ffffffff8d432326&gt;] SyS_unlink+0x16/0x20 [11497.552753] [&lt;ffffffff8d92579b&gt;] system_call_fastpath+0x22/0x27 [11497.559484] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [11497.565069] Kernel panic - not syncing: LBUG [11497.569837] CPU: 7 PID: 132407 Comm: spades Kdump: loaded Tainted: G OE ------------ 3.10.0-862.14.4.el7.x86_64 #1 [11497.582928] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.8.0 005/17/2018 [11497.591379] Call Trace: [11497.594105] [&lt;ffffffff8d913754&gt;] dump_stack+0x19/0x1b [11497.599845] [&lt;ffffffff8d90d29f&gt;] panic+0xe8/0x21f [11497.605211] [&lt;ffffffffc09948cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [11497.612131] [&lt;ffffffffc0f25c87&gt;] ll_delete_inode+0x1b7/0x1c0 [lustre] [11497.619421] [&lt;ffffffff8d43c504&gt;] evict+0xb4/0x180 [11497.624775] [&lt;ffffffff8d43ce0c&gt;] iput+0xfc/0x190 [11497.630033] [&lt;ffffffff8d43126e&gt;] do_unlinkat+0x1ae/0x2d0 [11497.636064] [&lt;ffffffff8d42175e&gt;] ? ____fput+0xe/0x10 [11497.641709] [&lt;ffffffff8d2bab90&gt;] ? task_work_run+0xc0/0xe0 [11497.647935] [&lt;ffffffff8d432326&gt;] SyS_unlink+0x16/0x20 [11497.653679] [&lt;ffffffff8d92579b&gt;] system_call_fastpath+0x22/0x27 </pre> </div></div> <p>Thanks,<br/> Stephane</p> Client: CentOS 7.5 Lustre 2.10.5 <br/> Server (Oak) CentOS 7.4 Lustre 2.10.4 LU-11582

LBUG: ASSERTION( inode->i_data.nrpages == 0 ) failed

Bug Critical Resolved Fixed Zhenyu Xu Stephane Thiell Mon, 29 Oct 2018 21:17:22 +0000 Tue, 5 Nov 2019 05:38:27 +0000 Tue, 27 Nov 2018 05:10:55 +0000 Lustre 2.10.5 Lustre 2.12.0 Lustre 2.10.7 0 9 <p>vmcore uploaded to your ftp server, the file is vmcore-sh-112-03-2018-10-26-22-17-26_<a href="https://jira.whamcloud.com/browse/LU-11582" title="LBUG: ASSERTION( inode-&gt;i_data.nrpages == 0 ) failed" class="issue-link" data-issue-key="LU-11582"><del>LU-11582</del></a></p> <p>kernel used is CentOS 7 3.10.0-862.14.4.el7.x86_64 ( <a href="http://debuginfo.centos.org/7/x86_64/" class="external-link" target="_blank" rel="nofollow noopener">http://debuginfo.centos.org/7/x86_64/</a> )</p> <p>Thanks!</p> <p>Bobijam</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>I think the assertion is reading the nrpages without supposed being protected under mapping-&gt;tree_lock, and truncate_inode_pages() is traverse the mapping's radix tree without tree_lock, and could miss finding the page being removed from the radix in __remove_mapping()</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>truncate_inode_pages_final()</b></div><div class="codeContent panelContent"> <pre class="code-java"> nrpages = mapping-&gt;nrpages; smp_rmb(); nrexceptional = mapping-&gt;nrexceptional; <span class="code-keyword">if</span> (nrpages || nrexceptional) { /* * As truncation uses a lockless tree lookup, cycle * the tree lock to make sure any ongoing tree * modification that does not see AS_EXITING is * completed before starting the <span class="code-keyword">final</span> truncate. */ spin_lock_irq(&amp;mapping-&gt;tree_lock); spin_unlock_irq(&amp;mapping-&gt;tree_lock); <span class="code-comment">// race window, that __remove_mapping() removes the page from the radix, </span> <span class="code-comment">// but nrpages hasn't been decreased yet. </span> truncate_inode_pages(mapping, 0); } </pre> </div></div> <p>And I think our truncate_inode_pages_final() in lustre/include/lustre_compat.h made the right sequence call</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #ifndef HAVE_TRUNCATE_INODE_PAGES_FINAL <span class="code-keyword">static</span> inline void truncate_inode_pages_final(struct address_space *map) { truncate_inode_pages(map, 0); <span class="code-comment">/* Workaround <span class="code-keyword">for</span> LU-118 */</span> <span class="code-keyword">if</span> (map-&gt;nrpages) { spin_lock_irq(&amp;map-&gt;tree_lock); <span class="code-comment">// after get the tree_lock, we avoid the race </span> spin_unlock_irq(&amp;map-&gt;tree_lock); } <span class="code-comment">/* Workaround end */</span> } #endif </pre> </div></div> <p>I think the fix could be add a tree_lock for checking the nrpages in ll_delete_inode, or just delete this assertion.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/33639" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33639</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11582" title="LBUG: ASSERTION( inode-&gt;i_data.nrpages == 0 ) failed" class="issue-link" data-issue-key="LU-11582"><del>LU-11582</del></a> llite: protect reading inode-&gt;i_data.nrpages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c9fb1cae878d4c3df24173e8b3c6b436fe983533</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/33681" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33681</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11582" title="LBUG: ASSERTION( inode-&gt;i_data.nrpages == 0 ) failed" class="issue-link" data-issue-key="LU-11582"><del>LU-11582</del></a> llite: protect reading inode-&gt;i_data.nrpages<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 35396125e071ad1457257348be38721bc9ffdad5</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33639/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33639/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11582" title="LBUG: ASSERTION( inode-&gt;i_data.nrpages == 0 ) failed" class="issue-link" data-issue-key="LU-11582"><del>LU-11582</del></a> llite: protect reading inode-&gt;i_data.nrpages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 04c172b686763be0d42eb4c36532d5795166eb7c</p> <p>Landed for 2.12</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33681/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33681/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11582" title="LBUG: ASSERTION( inode-&gt;i_data.nrpages == 0 ) failed" class="issue-link" data-issue-key="LU-11582"><del>LU-11582</del></a> llite: protect reading inode-&gt;i_data.nrpages<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: d936fdca9e00e05cecd4b21c6e4bbbf7107dc9b4</p> Duplicate Related LU-118 Development Rank 1|i005av: Rank (Obsolete) 9223372036854775807 Severity