[LU-11373] CERROR/CWARN messages are not throttled

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.12.0
• Affects Version/s: Lustre 2.12.0
• Assignee: Andreas Dilger
• Reporter: Andreas Dilger

Description

It appears that CDEBUG_LIMIT is not working properly, since test logs in Maloo are full of repeated error messages. This would be a nightmare for a large cluster if there are many clients and servers spewing repeated messages.

For example, sanity test_60a on the MDS console:

[ 2786.154265] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.155427] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.156482] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.157628] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.158671] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.159789] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.160824] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.161934] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.162977] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.164074] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[ 2786.165111] Lustre: 30107:0:(llog_cat.c:98:llog_cat_new_log()) MGS: there are no more free slots in catalog e973af35
[repeats hundreds of times]

It might relate to ktime_t patches that James landed previously, but that is just speculation as I haven't investigated it yet.

The sanity test_60b should be catching the failure of CDEBUG_LIMIT() but it is checking the logs on the client, while the test is being run on the MGS.

Issue Links

• is duplicated by: LU-11384 wrong check in libcfs_debug_vmsg2()
• is duplicated by: LU-11383 A lot of cfs_fail_loc error message for Fake I/O
• is related to: LU-11566 sanity test_60aa: llog_print_cb()) not enough space for print log records

Comments

[Andreas Dilger]

Any patches failing due to this issue need to rebase to get the fix.

[Peter Jones]

Landed for 2.12

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33474/
Subject: LU-11373 tests: increase debug limit sanity 60b
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4afee32188d38d2969ea160ddec64cd8359e12ca

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33474
Subject: LU-11373 tests: increase debug limit sanity 60b
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8a6f5af4b66815f97e7437a1d4d98b49f88a43e2

[James Nunez (Inactive)]

Looking at the failures so far, it looks like ZFS is the factor in these failures. Looking at all sanity test 60b from the past 3 days, none of the ldiskfs test sessions, including DNE test sessions, had any failures.

All failures are for ZFS test sessions.

It looks like all the failures fail with 102 or 103 debug lines. Although all of these failures count two extra lines from test_60aa (LU-11566), I'll submit a patch to increase the check from 100 to 103.

[Andreas Dilger]

It looks like the failure is with 103 debug messages vs. expected 100 messages. I can expect DNE is more verbose (it looks like the other MDTs are still mounted and complaining), so bumping this limit up a bit is fine.