https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16637 Lustre <p>truncate_inode_pages() is required to be called under (and serialised by) inode lock, or else new page could be added in the inode's page tree even ll_truncate_inode_pages_final() leverages mapping-&gt;i_pages protection to prevent a page is still in the process of deletion (inside __delete_from_page_cache())</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>649346.462967] LustreError: 482172:0:(llite_lib.c:2235:ll_truncate_inode_pages_final()) ASSERTION( nrpages == 0 ) failed: exa5: inode=[0x5c004247c:0x9b:0x0](000000000c1d2c09) nrpages=1, see https://jira.whamcloud.com/browse/LU-118 [649346.483202] LustreError: 482172:0:(llite_lib.c:2235:ll_truncate_inode_pages_final()) LBUG [649346.491472] Pid: 482172, comm: nsmb6.09.23.mpi 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Wed Jan 19 17:53:40 UTC 2022 [649346.501648] Call Trace TBD: [649346.504644] [&lt;0&gt;] libcfs_call_trace+0x6f/0x90 [libcfs] [649346.509880] [&lt;0&gt;] lbug_with_loc+0x43/0x80 [libcfs] [649346.514891] [&lt;0&gt;] ll_truncate_inode_pages_final+0xa7/0xe0 [lustre] [649346.521185] [&lt;0&gt;] vvp_prune+0x178/0x1d0 [lustre] [649346.525932] [&lt;0&gt;] cl_object_prune+0x51/0x110 [obdclass] [649346.531263] [&lt;0&gt;] lov_conf_set+0x5ad/0xa90 [lov] [649346.536007] [&lt;0&gt;] cl_conf_set+0x59/0x110 [obdclass] [649346.540995] [&lt;0&gt;] ll_layout_conf+0x13e/0x3d0 [lustre] [649346.546160] [&lt;0&gt;] ll_layout_refresh+0x581/0x980 [lustre] [649346.551588] [&lt;0&gt;] vvp_io_init+0x225/0x370 [lustre] [649346.556502] [&lt;0&gt;] cl_io_init0.isra.15+0x84/0x130 [obdclass] [649346.562186] [&lt;0&gt;] ll_fault_io_init+0x1cf/0x430 [lustre] [649346.567527] [&lt;0&gt;] ll_fault+0x222/0x900 [lustre] [649346.572158] [&lt;0&gt;] __do_fault+0x38/0xb0 [649346.576000] [&lt;0&gt;] do_fault+0x1a0/0x3c0 [649346.579851] [&lt;0&gt;] __handle_mm_fault+0x4d5/0x820 [649346.584473] [&lt;0&gt;] handle_mm_fault+0xbe/0x1e0 [649346.588836] [&lt;0&gt;] __do_page_fault+0x1ed/0x4c0 [649346.593289] [&lt;0&gt;] do_page_fault+0x37/0x130 [649346.597478] [&lt;0&gt;] page_fault+0x1e/0x30 [649346.601326] Kernel panic - not syncing: LBUG </pre> </div></div> LU-16637

ll_truncate_inode_pages_final() ASSERTION( nrpages == 0 ) failed: ... nrpages=1

Bug Major Resolved Fixed Zhenyu Xu Zhenyu Xu Mon, 13 Mar 2023 15:29:09 +0000 Fri, 26 Jan 2024 17:15:36 +0000 Tue, 23 Jan 2024 13:32:13 +0000 Lustre 2.16.0 0 9 <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50284" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50284</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16637" title="ll_truncate_inode_pages_final() ASSERTION( nrpages == 0 ) failed: ... nrpages=1" class="issue-link" data-issue-key="LU-16637"><del>LU-16637</del></a> llite: call truncate_inode_pages() under inode lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8d7eba935a6117e3afa8dd951358172fb14f08b7</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50284/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50284/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16637" title="ll_truncate_inode_pages_final() ASSERTION( nrpages == 0 ) failed: ... nrpages=1" class="issue-link" data-issue-key="LU-16637"><del>LU-16637</del></a> llite: call truncate_inode_pages() under inode lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ef9be34478036db0544753e33030fff7e32bfe44</p> <p>Noted that in a later kernel (v5.14) commit, and invalidate_lock was added to protect adding page to page cache, we might need to adapt that in new EL version.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 730633f0b7f951726e87f912a6323641f674ae34 Author: Jan Kara &lt;jack@suse.cz&gt; AuthorDate: Thu Jan 28 19:19:45 2021 +0100 Commit: Jan Kara &lt;jack@suse.cz&gt; CommitDate: Tue Jul 13 13:14:27 2021 +0200 mm: Protect operations adding pages to page cache with invalidate_lock Currently, serializing operations such as page fault, read, or readahead against hole punching is rather difficult. The basic race scheme is like: fallocate(FALLOC_FL_PUNCH_HOLE) read / fault / .. truncate_inode_pages_range() &lt;create pages in page cache here&gt; &lt;update fs block mapping and free blocks&gt; Now the problem is in this way read / page fault / readahead can instantiate pages in page cache with potentially stale data (if blocks get quickly reused). Avoiding this race is not simple - page locks do not work because we want to make sure there are *no* pages in given range. inode-&gt;i_rwsem does not work because page fault happens under mmap_sem which ranks below inode-&gt;i_rwsem. Also using it for reads makes the performance for mixed read-write workloads suffer. So create a new rw_semaphore in the address_space - invalidate_lock - that protects adding of pages to page cache for page faults / reads / readahead. Reviewed-by: Darrick J. Wong &lt;djwong@kernel.org&gt; Reviewed-by: Christoph Hellwig &lt;hch@lst.de&gt; Signed-off-by: Jan Kara &lt;jack@suse.cz&gt; </pre> </div></div> <p>Bobijam</p> <p>So does that mean that RHEL9.x clients would require a different fix?</p> <p>Peter</p> <p>I guess so, truncate_inode_pages need to acquire invalidate_lock to protect from pages cache filling (fault, read, ...) in kernel &gt; v5.14</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=qian_wc" class="user-hover" rel="qian_wc">qian_wc</a> , do your patches with the invalidate_lock cover this case as well?</p> <p>Hi,</p> <p>I originally thought that we need invalidate_lock in the following case:<br/> There are four place that will delete a page from Lustre:<br/> 1. extent lock blocking AST callback<br/> 2. truncate<br/> 3. punch hole<br/> 4. layout change will prune the pages</p> <p>Then I wrote test scripts to verify it, but found that:<br/> 1. Extent lock blocking AST callback: we solve it in the ticket <a href="https://jira.whamcloud.com/browse/LU-16651" title="hold invalidate_lock when invalidating page cache under kernel 5.15+" class="issue-link" data-issue-key="LU-16651"><del>LU-16651</del></a> llite: hold invalidate_lock when invalidate cache pages;<br/> 2. Truncate: For user space truncate() operation, we hole <span class="error">&#91;size, EOF&#93;</span> extent lock for the truncate() operations, thus it is same problem as Case 1.<br/> 3. punch hole: Lustre currently does not support it.<br/> 4. layout change will prune the pages: we can only change layout when there is no I/O activities (@&amp;lov-&gt;lo_active_ios == 0), so this can also not happen.</p> <p>So I think the patch under <a href="https://jira.whamcloud.com/browse/LU-16651" title="hold invalidate_lock when invalidating page cache under kernel 5.15+" class="issue-link" data-issue-key="LU-16651"><del>LU-16651</del></a> is enough.<br/> It does not need to hold invalidate_lock in this case for the current Lustre, I think.</p> <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50857" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50857</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16637" title="ll_truncate_inode_pages_final() ASSERTION( nrpages == 0 ) failed: ... nrpages=1" class="issue-link" data-issue-key="LU-16637"><del>LU-16637</del></a> llite: call truncage_inode_pages() in inode lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 94d493f7b99eaa681628f068e8d18979e99fdb26</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50857/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50857/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16637" title="ll_truncate_inode_pages_final() ASSERTION( nrpages == 0 ) failed: ... nrpages=1" class="issue-link" data-issue-key="LU-16637"><del>LU-16637</del></a> llite: call truncate_inode_pages() in inode lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 51d62f2122fee14fbb3ff8333b5a830e1181e4e5</p> <p>Landed for 2.16</p> <p>I'm able to reproduce the assertion failure with the fix in place. </p> <p>The reproducer is the following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> swap() {         cd /mnt/lustre         <span class="code-keyword">while</span> <span class="code-keyword">true</span>; <span class="code-keyword">do</span>                 lfs swap_layouts f1 f2         done } writedata() {         <span class="code-keyword">while</span> <span class="code-keyword">true</span>; <span class="code-keyword">do</span>                 echo -n a         done &gt; /mnt/lustre/f1 } touch /mnt/lustre/f1 touch /mnt/lustre/f2 swap &amp; writedata &amp; writedata &amp; writedata &amp; writedata &amp; writedata &amp; writedata &amp; writedata &amp; writedata &amp; wait </pre> </div></div> <p>The following change was made to Lustre hash c1ebcb0fa to speed up reproduction:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/llite/llite_lib.c b/lustre/llite/llite_lib.c index eba1cf4ff51..9542fb98e97 100644 --- a/lustre/llite/llite_lib.c +++ b/lustre/llite/llite_lib.c @@ -3011,6 +3011,8 @@ void ll_truncate_inode_pages_final(struct inode *inode)           truncate_inode_pages_final(mapping);   +       udelay(10000); +         /* Workaround <span class="code-keyword">for</span> LU-118: Note nrpages may not be totally updated when          * truncate_inode_pages() returns, as there can be a page in the process          * of deletion (inside __delete_from_page_cache()) in the specified diff --git a/lustre/utils/lfs.c b/lustre/utils/lfs.c index f3f8101dadc..8fd223a7b1a 100644 --- a/lustre/utils/lfs.c +++ b/lustre/utils/lfs.c @@ -10765,8 +10765,8 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> lfs_swap_layouts(<span class="code-object">int</span> argc, <span class="code-object">char</span> **argv)                 <span class="code-keyword">return</span> CMD_HELP;           <span class="code-keyword">return</span> llapi_swap_layouts(argv[1], argv[2], 0, 0, -                                 SWAP_LAYOUTS_KEEP_MTIME | -                                 SWAP_LAYOUTS_KEEP_ATIME); +                                 /* SWAP_LAYOUTS_KEEP_MTIME | +                                 SWAP_LAYOUTS_KEEP_ATIME*/ 0);  }    <span class="code-keyword">static</span> <span class="code-keyword">const</span> <span class="code-object">char</span> *<span class="code-keyword">const</span> ladvise_names[] = LU_LADVISE_NAMES; </pre> </div></div> <p>After a few minutes Lustre panics with:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 2104.190146] LustreError: 24712:0:(llite_lib.c:3035:ll_truncate_inode_pages_final()) ASSERTION( nrpages == 0 ) failed: lustre: inode=[0x200000401:0x1:0x0](000000007d0affed) nrpages=1 state 0x5, lli_flags 0x15, see https:<span class="code-comment">//jira.whamcloud.com/browse/LU-118 </span>[ 2104.458673] LustreError: 24712:0:(llite_lib.c:3035:ll_truncate_inode_pages_final()) LBUG [ 2104.555802] Kernel panic - not syncing: LBUG [ 2104.606854] CPU: 6 PID: 24712 Comm: bash Kdump: loaded Tainted: G           OE    --------- -  - 4.18.0-305.10.2.x6.1.010.21.x86_64 #1 [ 2104.751515] Hardware name: Gigabyte Technology Co., Ltd. P55-US3L/P55-US3L, BIOS FE 03/05/2010 [ 2104.854787] Call Trace: [ 2104.884121]  dump_stack+0x5c/0x80 [ 2104.923848]  panic+0xe7/0x2a9 [ 2104.959422]  ? entry_SYSCALL_64_after_hwframe+0x65/0xca [ 2105.021930]  lbug_with_loc+0x85/0x90 [libcfs] [ 2105.074052]  ll_truncate_inode_pages_final+0xd5/0x170 [lustre] [ 2105.143927]  vvp_prune+0x1c6/0x2a0 [lustre] [ 2105.193978]  cl_object_prune+0x72/0x140 [obdclass] [ 2105.251457]  lov_conf_set+0x1c7/0xde0 [lov] [ 2105.301772]  ? batch_send_update_req.constprop.8+0x381/0x8a0 [ptlrpc] [ 2105.378870]  cl_conf_set+0x7a/0x150 [obdclass] [ 2105.432215]  ll_layout_conf+0x8b/0x420 [lustre] [ 2105.486467]  ? lock_res_and_lock+0x29/0x50 [ptlrpc] [ 2105.544846]  ? ll_layout_refresh+0x79b/0xa20 [lustre] [ 2105.605379]  ll_layout_refresh+0x79b/0xa20 [lustre] [ 2105.663819]  vvp_io_init+0x22a/0x400 [lustre] [ 2105.716041]  __cl_io_init.isra.13+0x9e/0x170 [obdclass] [ 2105.778737]  ll_file_io_generic+0x317/0xdf0 [lustre] [ 2105.838228]  ll_file_write_iter+0x5d4/0x8d0 [lustre] [ 2105.897596]  ? __switch_to_asm+0x35/0x70 [ 2105.944499]  ? __switch_to_asm+0x41/0x70 [ 2105.991419]  new_sync_write+0x112/0x160 [ 2106.037378]  vfs_write+0xa5/0x1a0 [ 2106.077099]  ksys_write+0x4f/0xb0 [ 2106.116730]  do_syscall_64+0x5b/0x1a0 [ 2106.160514]  entry_SYSCALL_64_after_hwframe+0x65/0xca [ 2106.220935] RIP: 0033:0x7f25853f5ba0 [ 2106.263778] Code: 73 01 c3 48 8b 0d d0 72 2d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d 1d d4 2d 00 00 75 10 b8 01 00 00 00 0f 05 &lt;48&gt; 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 7e cc 01 00 48 89 04 24 [ 2106.489251] RSP: 002b:00007ffdb6797928 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 2106.580144] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f25853f5ba0 [ 2106.665733] RDX: 0000000000000001 RSI: 00007f2585d19000 RDI: 0000000000000001 [ 2106.751223] RBP: 00007f2585d19000 R08: 00007f2585d06740 R09: 00007f25853532cd [ 2106.836919] R10: 00007f2585d06740 R11: 0000000000000246 R12: 00007f25856ce400 [ 2106.922614] R13: 0000000000000001 R14: 0000000000000000 R15: 00000000026364c2 </pre> </div></div> <p>As I wrote in the gerrit review, non-zero page count after truncate is likely the result of<br/> a read/write/fault running in parallel with truncate. Most obviously, in a write test tiny<br/> writes add pages to the page cache even though they cannot initialize and run I/O on them.</p> <p>It seems, one of the possible solutions is to follow the approach from 9c453ba6d9. Make sure truncates are wrapped with trunc_sem in exclusive mode and reads, writes and faults including their fast paths are wrapped with trunc sem in shared mode.</p> <p>Hi Andrew,</p> <p>Is it possibly better to loose the assertion condition to (LASSERTF(nrpages == 0 || nrpages == 1, ...)? I don't know whether this window would allow more than 1 page be inserted in the page cache, but just observed that all these cases shows only 1 page is in the page cache after tripping the assertion failure.</p> <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53316" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53316</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16637" title="ll_truncate_inode_pages_final() ASSERTION( nrpages == 0 ) failed: ... nrpages=1" class="issue-link" data-issue-key="LU-16637"><del>LU-16637</del></a> llite: page in cache after truncate_inode_pages()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e67690e9da79362787c7bf959823fcfe0604dcd8</p> <p>@bobijam,</p> <p> </p> <p>I think there can be any number of pages, since you can have any number of parallel read requests, each of them can add a new cache page.</p> <p>"Andrew Perepechko &lt;andrew.perepechko@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53554" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53554</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16637" title="ll_truncate_inode_pages_final() ASSERTION( nrpages == 0 ) failed: ... nrpages=1" class="issue-link" data-issue-key="LU-16637"><del>LU-16637</del></a> llite: tolerate fresh page cache pages after truncate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e9aade50eee736299d08405c1c9e3fee5b5fa8f7</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53554/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53554/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16637" title="ll_truncate_inode_pages_final() ASSERTION( nrpages == 0 ) failed: ... nrpages=1" class="issue-link" data-issue-key="LU-16637"><del>LU-16637</del></a> llite: tolerate fresh page cache pages after truncate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7bb1e211d217d5a82ac2d5e4edad5ae018090761</p> <p>All patches merged for 2.16</p> Related LU-16958 Development Rank 1|i03g5b: Rank (Obsolete) 9223372036854775807 Severity