[LU-5857] catastrophe cannot be found: “error: get_param: /proc/{fs,sys}/{lnet,lustre}/catastrophe: Found no match”

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.7.0
• Affects Version/s: Lustre 2.7.0
• Environment: Autotest
• Severity: 3
• Assignee: Jian Yu
• Reporter: James Nunez (Inactive)

Description

In conf-sanity, each test is called from a wrapper function run_test(). When the test completes, run_test() eventually calls check_catastrophe. For some tests, check_catastrophe() does not complete correctly:

02:31:40:CMD: onyx-38vm3,onyx-38vm4,onyx-38vm5 rc=\$(lctl get_param -n catastrophe);
02:31:40:		if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi
02:31:40:		exit \$rc
02:31:40:onyx-38vm5: error: get_param: /proc/{fs,sys}/{lnet,lustre}/catastrophe: Found no match
02:31:40:onyx-38vm5: sh: line 1: [: -ne: unary operator expected

For example, conf-sanity tests 53a, 53b, 56, 57a, 58, 61, 62, 66, 67 and others at https://testing.hpdd.intel.com/test_sets/5634cf00-6375-11e4-b5da-5254006e85c2 have these errors during the test clean up in the call to check_catastrophe.

More of these error messages in conf-sanity can be found at:
https://testing.hpdd.intel.com/test_sets/20be82f6-637c-11e4-80e1-5254006e85c2

The test is correctly marked as pass or fail regardless of if check_catastrophe works or not.

Comments

[Jodi Levi (Inactive)]

Patch landed to Master.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12640/
Subject: LU-5857 tests: check lctl return value in check_catastrophe()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 43d19e90c719402a8d73e559ce4368aa55a4f16b

[Jian Yu]

Patch for master branch: http://review.whamcloud.com/12640

[Jian Yu]

I'll upload a patch to improve check_catastrophe() accordingly.

[Andreas Dilger]

In that case, this shouldn't even be done if the libcfs module isn't loaded, since it isn't possible to have an LBUG and then unload the modules.

[Jian Yu]

unload_modules() was called at the end of those sub-tests before running check_catastrophe().

[Andreas Dilger]

Is this failing because the filesystem is unmounted and the modules removed when get_param is called, or is there some problem with this file in /proc?

[Jodi Levi (Inactive)]

Yu Jian,
Could you please look into this one?
Thank you!