https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11373 Lustre <p>It appears that CDEBUG_LIMIT is not working properly, since test logs in Maloo are full of repeated error messages. This would be a nightmare for a large cluster if there are many clients and servers spewing repeated messages.</p> <p>For example, sanity test_60a on the MDS console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2786.154265] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.155427] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.156482] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.157628] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.158671] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.159789] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.160824] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.161934] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.162977] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.164074] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [ 2786.165111] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35 [repeats hundreds of times] </pre> </div></div> <p>It <em>might</em> relate to ktime_t patches that James landed previously, but that is just speculation as I haven't investigated it yet.</p> <p>The sanity <tt>test_60b</tt> should be catching the failure of <tt>CDEBUG_LIMIT()</tt> but it is checking the logs on the client, while the test is being run on the MGS.</p> LU-11373

CERROR/CWARN messages are not throttled

Bug Critical Resolved Fixed Andreas Dilger Andreas Dilger Wed, 12 Sep 2018 21:52:51 +0000 Wed, 31 Oct 2018 06:15:35 +0000 Tue, 30 Oct 2018 03:54:36 +0000 Lustre 2.12.0 Lustre 2.12.0 0 3 <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33154" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33154</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11373" title="CERROR/CWARN messages are not throttled" class="issue-link" data-issue-key="LU-11373"><del>LU-11373</del></a> tests: fix sanity test_60b to run on the MGS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1c37077c2d5340a56914908e5e3f469b3d1d8394</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33154/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33154/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11373" title="CERROR/CWARN messages are not throttled" class="issue-link" data-issue-key="LU-11373"><del>LU-11373</del></a> libcfs: fix wrong check in libcfs_debug_vmsg2()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4037c1462730c120af00bd0e95b08fe0669e0271</p> <p>Landed for 2.12</p> <p>Just in case, I have just experienced an Oops due to soft-lockup (i.e. with /proc/sys/kernel/softlockup_panic set to 1) where my crash-dump analysis (all cores but one stuck in stop_machine()/stop_cpu() during module unload, when the last core seems to print Console msgs since too long time, right after sanity/test_60a/llog_test_10() and due to catalog full simulation) has permitted me to also identify that the root cause could be this Console msgs rate limitation failure/regression!</p> <p>But just in case a working rate limiting will not be enough to fix, I also wonder if the number of LLOG record creation/write attempts could be slightly reduced in sanity/test_60a/llog_test_10() and I have also found a specific module-unload/stop_machine() scenario and fix report at <a href="https://lists.openwall.net/linux-kernel/2015/08/19/367" class="external-link" target="_blank" rel="nofollow noopener">https://lists.openwall.net/linux-kernel/2015/08/19/367</a> which could be of interest.</p> <p> </p> <p>Reopened ticket because we have at least two recent sanity test 60b failures after the patch for this ticket was landed; <a href="https://testing.whamcloud.com/test_sets/e51fa37e-d6db-11e8-ad90-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e51fa37e-d6db-11e8-ad90-52540065bddc</a> and <a href="https://testing.whamcloud.com/test_sets/45b22ad4-d6d4-11e8-ad90-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/45b22ad4-d6d4-11e8-ad90-52540065bddc</a> .</p> <p>It looks like the failure is with 103 debug messages vs. expected 100 messages. I can expect DNE is more verbose (it looks like the other MDTs are still mounted and complaining), so bumping this limit up a bit is fine. </p> <p>Looking at the failures so far, it looks like ZFS is the factor in these failures. Looking at all sanity test 60b from the past 3 days, none of the ldiskfs test sessions, including DNE test sessions, had any failures. </p> <p>All failures are for ZFS test sessions. </p> <p>It looks like all the failures fail with 102 or 103 debug lines. Although all of these failures count two extra lines from test_60aa (<a href="https://jira.whamcloud.com/browse/LU-11566" title="sanity test_60aa: llog_print_cb()) not enough space for print log records" class="issue-link" data-issue-key="LU-11566"><del>LU-11566</del></a>), I'll submit a patch to increase the check from 100 to 103. </p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33474" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33474</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11373" title="CERROR/CWARN messages are not throttled" class="issue-link" data-issue-key="LU-11373"><del>LU-11373</del></a> tests: increase debug limit sanity 60b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8a6f5af4b66815f97e7437a1d4d98b49f88a43e2</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33474/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33474/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11373" title="CERROR/CWARN messages are not throttled" class="issue-link" data-issue-key="LU-11373"><del>LU-11373</del></a> tests: increase debug limit sanity 60b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4afee32188d38d2969ea160ddec64cd8359e12ca</p> <p>Landed for 2.12</p> <p>Any patches failing due to this issue need to rebase to get the fix.</p> Duplicate LU-11384 LU-11383 Related LU-11566 Development Rank 1|i002bz: Rank (Obsolete) 9223372036854775807