https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13063 Lustre <p>The last thing seen in the suite_log for sanity test 411 is </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 411: Slab allocation error with cgroup does not LBUG ================================== 04:54:33 (1575953673) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.88888 s, 27.0 MB/s </pre> </div></div> <p>Normally, on successful runs, we would see a dd error reading the file just created, but the test hangs at this point. Looking at the console logs, it’s not clear why the test is hanging, but we see lnet-selftest processes hung. Looking at the stack trace on the first client (vm10), we see that there a lnet-selftest process stuck D state</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[14127.185129] lst_t_00_00 S 0 14488 2 0x80000080 [14127.186075] Call Trace: [14127.186561] ? __schedule+0x253/0x830 [14127.187236] ? sfw_test_unit_done.isra.14+0x9d/0x150 [lnet_selftest] [14127.188348] schedule+0x28/0x70 [14127.188929] cfs_wi_scheduler+0x40d/0x420 [libcfs] [14127.189783] ? finish_wait+0x80/0x80 [14127.190466] ? cfs_wi_sched_create+0x5a0/0x5a0 [libcfs] [14127.191397] kthread+0x112/0x130 [14127.191984] ? kthread_flush_work_fn+0x10/0x10 [14127.192782] ret_from_fork+0x35/0x40 [14127.193448] st_timer D 0 14636 2 0x80000080 [14127.194413] Call Trace: [14127.194882] ? __schedule+0x253/0x830 [14127.195555] schedule+0x28/0x70 [14127.196142] schedule_timeout+0x16b/0x390 [14127.196859] ? __next_timer_interrupt+0xc0/0xc0 [14127.197678] ? prepare_to_wait_event+0xbb/0x140 [14127.198496] stt_timer_main+0x215/0x230 [lnet_selftest] [14127.199436] ? finish_wait+0x80/0x80 [14127.200083] ? sfw_startup+0x540/0x540 [lnet_selftest] [14127.200989] kthread+0x112/0x130 [14127.201595] ? kthread_flush_work_fn+0x10/0x10 [14127.202393] ret_from_fork+0x35/0x40 </pre> </div></div> <p>Similarly in the stack-trace log on the MDS (vm12), we see the lnet process</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[14034.774700] st_timer D ffff9cb15b62a080 0 28114 2 0x00000080 [14034.776068] Call Trace: [14034.776493] [&lt;ffffffffb0f6af19&gt;] schedule+0x29/0x70 [14034.777425] [&lt;ffffffffb0f68968&gt;] schedule_timeout+0x168/0x2d0 [14034.778391] [&lt;ffffffffb08cfeb4&gt;] ? __wake_up+0x44/0x50 [14034.779358] [&lt;ffffffffb08aab30&gt;] ? __internal_add_timer+0x130/0x130 [14034.780432] [&lt;ffffffffb08c3a46&gt;] ? prepare_to_wait+0x56/0x90 [14034.781474] [&lt;ffffffffc1542a98&gt;] stt_timer_main+0x168/0x220 [lnet_selftest] [14034.782654] [&lt;ffffffffb08c3f50&gt;] ? wake_up_atomic_t+0x30/0x30 [14034.783688] [&lt;ffffffffc1542930&gt;] ? sfw_startup+0x580/0x580 [lnet_selftest] [14034.784856] [&lt;ffffffffb08c2e81&gt;] kthread+0xd1/0xe0 [14034.785787] [&lt;ffffffffb08c2db0&gt;] ? insert_kthread_work+0x40/0x40 [14034.786818] [&lt;ffffffffb0f77c37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [14034.788077] [&lt;ffffffffb08c2db0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>lnet-selftest did run and fail (<a href="https://jira.whamcloud.com/browse/LU-10073" title="lnet-selftest test_smoke: lst Error found" class="issue-link" data-issue-key="LU-10073"><del>LU-10073</del></a>) previous to sanity running. It’s not clear if lnet-selftest is a cause of this test hang.</p> <p>We’ve see this test hang twice for RHEL 8.1 testing both in December<br/> <a href="https://testing.whamcloud.com/test_sets/293b5216-1b13-11ea-a9d7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/293b5216-1b13-11ea-a9d7-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/133daa46-1b8a-11ea-b1e8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/133daa46-1b8a-11ea-b1e8-52540065bddc</a></p> <p>In addition, we've seen this once in the past 3 months in PPC testing for a patch for <a href="https://jira.whamcloud.com/browse/LU-11997" title="Crash in lustre_swab_fiemap" class="issue-link" data-issue-key="LU-11997"><del>LU-11997</del></a> at <a href="https://testing.whamcloud.com/test_sets/b4851392-f175-11e9-b62b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b4851392-f175-11e9-b62b-52540065bddc</a> .</p> RHEL 8.1 LU-13063

sanity test 411 times out for RHEL8.1

Bug Minor Open Unresolved Dongyang Li James Nunez always_except rhel8 Wed, 11 Dec 2019 17:13:01 +0000 Wed, 29 Mar 2023 18:04:02 +0000 Lustre 2.14.0 Lustre 2.12.4 0 10 <p>I have a patch that replaces cfs_wi_schedular with a workqueue. Give me a bit to get the patch working and I will push it.</p> <p>The patch looks like <a href="https://review.whamcloud.com/36991" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36991</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-9859" title="libcfs simplification" class="issue-link" data-issue-key="LU-9859">LU-9859</a> lnet: convert selftest to use workqueues</tt>"</p> <p>sanity test 411 still hangs for RHEL 8.1 when rebased on top of James's patch <a href="https://review.whamcloud.com/#/c/36991/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/36991/</a>. Results at <a href="https://testing.whamcloud.com/test_sets/ca87349e-21d2-11ea-80b4-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ca87349e-21d2-11ea-80b4-52540065bddc</a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dongyang" class="user-hover" rel="dongyang">dongyang</a> could you please advise here?</p> <p>from one of the test logs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [14170.962056] dd R running task 0 27130 26934 0x00000080 [14170.963279] Call Trace: [14170.963757] ? finish_task_switch+0x76/0x2b0 [14170.964525] ? __schedule+0x25b/0x830 [14170.965207] ? shrink_page_list+0x48e/0xc50 [14170.965970] ? page_evictable+0xe/0x40 [14170.966675] ? putback_inactive_pages+0x1f8/0x4f0 [14170.967517] ? shrink_inactive_list+0x207/0x570 [14170.968339] ? inactive_list_is_low+0xe0/0x220 [14170.969148] ? shrink_node_memcg+0x204/0x770 [14170.969920] ? shrink_node+0xce/0x440 [14170.970589] ? do_try_to_free_pages+0xc3/0x360 [14170.971396] ? try_to_free_mem_cgroup_pages+0xf9/0x210 [14170.972316] ? try_charge+0x192/0x780 [14170.972992] ? mem_cgroup_commit_charge+0x7a/0x560 [14170.973856] ? mem_cgroup_try_charge+0x8b/0x1a0 [14170.974677] ? __add_to_page_cache_locked+0x64/0x240 [14170.975560] ? add_to_page_cache_lru+0x4a/0xc0 [14170.976366] ? pagecache_get_page+0xf2/0x2c0 [14170.977294] ? ll_read_ahead_pages+0x1e8/0x8b0 [lustre] [14170.978305] ? osc_io_fini+0x10/0x10 [osc] [14170.979072] ? ll_readahead.constprop.32+0x641/0x9d0 [lustre] [14170.980105] ? ll_io_read_page+0x355/0x4c0 [lustre] [14170.980992] ? ll_readpage+0xe3/0x650 [lustre] [14170.981801] ? find_get_entry+0x19/0xf0 [14170.982508] ? pagecache_get_page+0x30/0x2c0 [14170.983286] ? generic_file_buffered_read+0x601/0xb10 [14170.984188] ? atime_needs_update+0x77/0xe0 [14170.984960] ? vvp_io_read_start+0x3ef/0x720 [lustre] [14170.986080] ? cl_lock_request+0x62/0x1b0 [obdclass] [14170.986989] ? cl_io_start+0x58/0x100 [obdclass] [14170.987851] ? cl_io_loop+0xdc/0x1b0 [obdclass] [14170.988682] ? ll_file_io_generic+0x23d/0x960 [lustre] [14170.989611] ? ll_file_read_iter+0x244/0x2e0 [lustre] [14170.990510] ? new_sync_read+0x121/0x170 [14170.991233] ? vfs_read+0x91/0x140 [14170.991864] ? ksys_read+0x4f/0xb0 [14170.992493] ? do_syscall_64+0x5b/0x1b0 [14170.993198] ? entry_SYSCALL_64_after_hwframe+0x65/0xca </pre> </div></div> <p>looks like dd is not <b>stuck</b>, we are reading a page and read_ahead kicks in, trying to allocate a page in page cache, hit cgroup limits and then it just keep trying to reclaim in a loop? </p> <p>Dongyang, in the case of a cgroup limit being hit, is there an error returned to the caller, or where is the allocation looping? It seems that <tt>ll_read_ahead_page()</tt> is already calling <tt>grab_cache_page_nowait()</tt> into <tt>pagecache_get_page(FGP_NOWAIT)</tt> so it <em>shouldn't</em> be blocking for pages that cannot be allocated. Maybe this is a bug in the kernel?</p> <p>Andreas, if we hit the cgroup limit in grab_cache_page_nowait(), it calls into mem_cgroup_try_charge() and eventually oom inside the cgroup.</p> <p>before doing that try_charge() will try to reclaim the pages, that's what we see here.</p> <p>yes ll_read_ahead_page() is using FGP_NOWAIT but that only means we won't wait for the</p> <p>page lock when getting the page ref. mapping-&gt;gfp_mask is still used by the cgroup reclaim.</p> <p>It does feels like a bug in the kernel but I'm having difficulty locating it, as well as reproducing it.</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37165" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37165</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13063" title="sanity test 411 times out for RHEL8.1" class="issue-link" data-issue-key="LU-13063">LU-13063</a> testing: reproduce sanity 411 error<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a0560ae8ebcc4559be7823ab0a5975e057979f4a</p> <blockquote> <p>mapping-&gt;gfp_mask is still used by the cgroup reclaim.</p></blockquote> <p>I was wondering about that. Are we sure that <tt>gfp&#95;mask</tt> excludes <tt>&#95;&#95;GFP&#95;FS</tt> at this point? The comment says this, but it isn't clear to me how that is working since it doesn't actually clear <tt>&#95;&#95;GFP&#95;FS</tt> that I can see:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> * Clear __GFP_FS when allocating the page to avoid recursion into the fs * and deadlock against the caller's locked page. */ <span class="code-keyword">static</span> inline struct page *grab_cache_page_nowait(struct address_space *mapping, pgoff_t index) { <span class="code-keyword">return</span> pagecache_get_page(mapping, index, FGP_LOCK|FGP_CREAT|FGP_NOFS|FGP_NOWAIT, mapping_gfp_mask(mapping)); } </pre> </div></div> <p>If we can reproduce this failure fairly easily, we might consider using our own version of this function that explicitly clears <tt>&#95;&#95;GFP&#95;FS</tt>, though this is just speculation (I don't actually see recursion in the stack):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* Clear __GFP_FS when allocating the page to avoid recursion into the fs * and deadlock against the caller's locked page. */ <span class="code-keyword">static</span> inline struct page *ll_grab_cache_page_nowait(struct address_space *mapping, pgoff_t index) { <span class="code-keyword">return</span> pagecache_get_page(mapping, index, FGP_LOCK|FGP_CREAT|FGP_NOFS|FGP_NOWAIT, mapping_gfp_mask(mapping) &amp; ~__GFP_FS); ] </pre> </div></div> <p>It would be worthwhile to see what has changed under <tt>linux/mm/</tt> between the RHEL 8.0 and 8.1 kernels to see if this would identify what is causing the failures.</p> <p>__GFP_FS is cleared in pagecache_get_page(), before we use the flag to alloc the page and add to lru,</p> <p>The comments from grab_cache_page_nowait() are misleading, see 45f87de57f8fad59302fd263dd81ffa4843b5b24</p> <p>grab_cache_page_nowait() passes FGP_NOFS to pagecache_get_page(), so we should be good.</p> <p>Can you please compare RHEL8.0 and 8.1 <tt>mm/</tt> tree and cgroups to see what has changed, since the test fails 100% on RHEL8.1 and never on 8.0, so there must be some code change in the kernel that is causing this. </p> <p>The changes are significant and I'm going through them. On the plus side, after enabling centos8-stream I'm able to get 4.18.0-147.3.1.el8_1.x86_64 and the newer 4.18.0-151.el8.x86_64. I can reproduce it on my vm on both kernels. and you are right it's a regression on 8.1, 8.0 is fine. The issue actually has nothing to do with lustre, can be reproduced doing dd from a local file.</p> <p>OK, turns out redhat is missing a patch for the 8.1 kernel.</p> <p>f9c645621a28e37813a1de96d9cbd89cde94a1e4 memcg, oom: don't require __GFP_FS when invoking memcg OOM killer</p> <p>how do we proceed? I can create a new 4.18-rhel8.1.series under kernel_patches and add the patch there, but that's for the server side, we need the kernel patch on the clients...</p> <p>It makes sense to add the patch to the RHEL8.1 series and we can test with the server packages installed on the client. </p> <p>As for opening a ticket with RedHat, I hope Oleg or Peter know the details on that.</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37197" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37197</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13063" title="sanity test 411 times out for RHEL8.1" class="issue-link" data-issue-key="LU-13063">LU-13063</a> kernel: fix try_charge retrying forever<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d4eae90c1ec4676f775bf7c0582c999968edf408</p> <p>How can we tell Maloo to use the server packages(where the kernel patch actually apples) on the clients?</p> <p>I looked into <a href="https://wiki.whamcloud.com/display/PUB/Changing+Test+Parameters+with+Gerrit+Commit+Messages" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.whamcloud.com/display/PUB/Changing+Test+Parameters+with+Gerrit+Commit+Messages</a></p> <p>but could not find anything related</p> <p>currently you can't install lustre server on client (unless you patched the client)</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37270" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37270</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13063" title="sanity test 411 times out for RHEL8.1" class="issue-link" data-issue-key="LU-13063">LU-13063</a> tests: stop running sanity test 411<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0681c9075cfb28745bdb90f660c36878a8b94679</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37272" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37272</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13063" title="sanity test 411 times out for RHEL8.1" class="issue-link" data-issue-key="LU-13063">LU-13063</a> tests: remove checks for old RHEL versions<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 734072a4ff70ff044fe44c3ff018f28720f38617</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37270/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37270/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13063" title="sanity test 411 times out for RHEL8.1" class="issue-link" data-issue-key="LU-13063">LU-13063</a> tests: stop running sanity test 411<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 34e4c37474b3d9328aac1dd9228a018ac7d4f47e</p> <p>The failure also occurred on Lustre b2_12 branch: <a href="https://testing.whamcloud.com/test_sets/0e3d7c8e-3ea7-11ea-9543-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0e3d7c8e-3ea7-11ea-9543-52540065bddc</a></p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37376" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37376</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13063" title="sanity test 411 times out for RHEL8.1" class="issue-link" data-issue-key="LU-13063">LU-13063</a> tests: fortestonly RHEL8.0 vs 8.1<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: d20b45141a917549858dc443c18c7b8b4f108ca8</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37272/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37272/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13063" title="sanity test 411 times out for RHEL8.1" class="issue-link" data-issue-key="LU-13063">LU-13063</a> tests: remove checks for old RHEL versions<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4a8e06fb5af88f93a936ed8ba0718ff5d3554c9f</p> <p>Jian Yu (yujian@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38260" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38260</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13063" title="sanity test 411 times out for RHEL8.1" class="issue-link" data-issue-key="LU-13063">LU-13063</a> tests: stop running sanity test 411<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: f857eb47ea816d4a74b0012049125ae8cbeb2149</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38260/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38260/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13063" title="sanity test 411 times out for RHEL8.1" class="issue-link" data-issue-key="LU-13063">LU-13063</a> tests: stop running sanity test 411<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 495a6f01a30855c72aa0ec4fda12f961342a97a3</p> <p>I don't think we're testing on RHEL 8.1 clients any more?  Are we?  I've removed this ALWAYS_EXCEPT as part of <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50460" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50460</a> ; hopefully I'm right about testing</p> Related LU-10073 Development Rank 1|i00quf: Rank (Obsolete) 9223372036854775807 Severity