[LU-5256] Test Infrastructure does not reset debug parameter

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1
• Type: Bug
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: WC Triage
• Resolution: Unresolved
• Labels: tests

Environment

All

• Severity: 3

Description

The debug parameter is overwritten in our test infrastructure and the attempts to return debug to its original value fail.

In cfg/local.sh, PTLDEBUG is set to

PTLDEBUG=${PTLDEBUG:-"vfstrace rpctrace dlmtrace neterror ha config ioctl super"}

If PTLDEBUG is set, the debug parameter is overwritten in routines in test-framework.sh. For example in load_modules_local(), we overwrite the debug parameter with

[ "$PTLDEBUG" ] && lctl set_param debug="$PTLDEBUG"

Running a tests with auster or calling the script directly does not change the situation

# lctl set_param debug=lfsck
debug=lfsck
# lctl get_param debug
debug=warning error emerg console lfsck

# NAME=local ONLY="1" ./sanity.sh
Logging to local directory: /tmp/test_logs/1403706713
c18: Checking config lustre mounted on /lustre/scratch
Checking servers environments
Checking clients c18 environments
...
== sanity test complete, duration 13 sec == 07:32:06 (1403706726)

# lctl get_param debug
debug=
super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console

# lctl set_param debug=lfsck
debug=lfsck
# lctl get_param debug
debug=warning error emerg console lfsck

# ./auster -f local -v -k sanity --only 2
Started at Wed Jun 25 07:46:57 PDT 2014
c18: Checking config lustre mounted on /lustre/scratch
Checking servers environments
Checking clients c18 environments
...
sanity returned 0
Finished at Wed Jun 25 07:47:13 PDT 2014 in 16s
./auster: completed with rc 0
# lctl get_param debug
debug=
super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console

Some test suites think they are returning debug to its original value, but they are thwarted by routines in test-framework setting the debug value before they can record and save the original value of debug. From sanity.sh, check_and_setup_lustre will set debug before this code is called

OLDDEBUG=$(lctl get_param -n debug 2> /dev/null)
lctl set_param debug=-1 2> /dev/null || true
...
if [ "$I_MOUNTED" != "yes" ]; then
	lctl set_param debug="$OLDDEBUG" 2> /dev/null || true
fi

Is there any reason debug shouldn't be set to "-1", log all types of messages, for all test suites? There will be more output, but this could be helpful to troubleshoot problems with tests.

Comments

[Andreas Dilger]

Running with full debug can slow down some if the tests significantly.