[LU-5251] lctl filter does not filter log messages

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.6.0
• Type: Improvement
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: WC Triage
• Resolution: Unresolved

Environment

OpenSFS cluster with Lustre 2.5.60 build #2535

• Severity: 3

Description

I'm having trouble filtering the kernel debug logs.The manual says that the command "lctl filter all_types" should remove all lines from the output of the debug log, but it does not filter anything.

For example, running the following on an MDS:

# lctl filter all_types
Disabling output of type "trace"
Disabling output of type "inode"
Disabling output of type "super"
Disabling output of type "ext2"
Disabling output of type "malloc"
Disabling output of type "cache"
Disabling output of type "info"
Disabling output of type "ioctl"
Disabling output of type "neterror"
Disabling output of type "net"
Disabling output of type "warning"
Disabling output of type "buffs"
Disabling output of type "other"
Disabling output of type "dentry"
Disabling output of type "nettrace"
Disabling output of type "page"
Disabling output of type "dlmtrace"
Disabling output of type "error"
Disabling output of type "emerg"
Disabling output of type "ha"
Disabling output of type "rpctrace"
Disabling output of type "vfstrace"
Disabling output of type "reada"
Disabling output of type "mmap"
Disabling output of type "config"
Disabling output of type "console"
Disabling output of type "quota"
Disabling output of type "sec"
Disabling output of type "lfsck"
Disabling output of type "hsm"
# lctl debug_kernel /tmp/log_lfsck_2
Debug log: 3850 lines, 3850 kept, 0 dropped, 0 bad.
# lctl debug_kernel /tmp/log_lfsck_2
Debug log: 255 lines, 255 kept, 0 dropped, 0 bad.
# lctl get_param debug
debug=
super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck

Some lines from the output should be dropped from the logs.

Maybe just the "all_types" flag is broken, but specifying individual types does not work:

# lctl filter vfstrace
Disabling output of type "vfstrace"
# lctl debug_kernel /tmp/log_lfsck_2
Debug log: 1743 lines, 1743 kept, 0 dropped, 0 bad.
# lctl debug_kernel /tmp/log_lfsck_2
Debug log: 66 lines, 66 kept, 0 dropped, 0 bad.
# lctl filter warning
Disabling output of type "warning"
# lctl filter ioctl
Disabling output of type "ioctl"
# lctl filter super
Disabling output of type "super"
# lctl debug_kernel /tmp/log_lfsck_2
Debug log: 765 lines, 765 kept, 0 dropped, 0 bad.
# lctl debug_kernel /tmp/log_lfsck_2
Debug log: 66 lines, 66 kept, 0 dropped, 0 bad.

No messages are dropped from the log output.

I tried using "lctl debug_daemon start" and lctl debug_daemon stop" and "lctl debug_file" to output and filter the logs, but no lines were filtered:

# lctl debug_daemon start /tmp/log_test1
# lctl filter all_types
Disabling output of type "trace"
Disabling output of type "inode"
...
# lctl show lfsck
Enabling output from subsystem "lfsck"
Enabling output of type "lfsck"
# lctl show trace
Enabling output of type "trace"
# lctl lfsck_start -M scratch-MDT0000 -type layout
Started LFSCK on the device scratch-MDT0000: scrub layout namespace
# lctl debug_daemon stop 
# lctl debug_file /tmp/log_test1 log_filtered
Debug log: 18532 lines, 18532 kept, 0 dropped, 0 bad.
[root@mds01 tmp]# lctl filter lfsck
Disabling output from subsystem "lfsck"
Disabling output of type "lfsck"
[root@mds01 tmp]# lctl debug_file /tmp/log_test1 log_filtered
Debug log: 18532 lines, 18532 kept, 0 dropped, 0 bad.

No lines are dropped even though I know there are lfsck messages in the log file /tmp/log_test1.

Comments

[Andreas Dilger]

I think this is just a usage problem, though possibly something that should be fixed in a way that makes lctl easier to use. The lctl command does not save the "filter" and "show" masks across lctl invocations. They are only really useful for interactive lctl use:

# lctl
lctl> filter all_types
lctl> show ldiskfs
lctl> df /tmp/debug /tmp/debug.txt

That said, this is not very convenient to use, and adding a "--filter" and "--show" options to debug_kernel may be useful.