[LU-2502] Test failure on test suite ost-pools, subtest test_23a

• Type: Bug
• Resolution: Won't Fix
• Priority: Critical
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.4, Lustre 2.4.1
• Labels: quota test yuc2
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for liuying <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/71f17b88-4826-11e2-8cdc-52540035b04c.

The sub-test test_23a failed with the following error:

test failed to respond and timed out

Client-1 console log showed

23:05:49:Lustre: DEBUG MARKER: == ost-pools test 23a: OST pools and quota =========================================================== 23:05:40 (1355727940)
23:05:49:Lustre: DEBUG MARKER: lctl get_param -n lov.lustre-*.pools.testpool 2>/dev/null || echo foo
23:05:49:Lustre: DEBUG MARKER: lctl get_param -n lov.lustre-*.pools.testpool | sort -u | tr '\n' ' ' 
23:06:01:Lustre: DEBUG MARKER: lctl get_param -n lov.lustre-*.pools.testpool | sort -u | tr '\n' ' ' 
23:06:01:LustreError: 19086:0:(quota_ctl.c:328:client_quota_ctl()) ptlrpc_queue_wait failed, rc: -114
23:06:01:LustreError: 19087:0:(lmv_obd.c:855:lmv_iocontrol()) error: iocontrol MDC lustre-MDT0000_UUID on MDTidx 0 cmd 800866a1: err = -22
23:06:01:LustreError: 19087:0:(lmv_obd.c:855:lmv_iocontrol()) error: iocontrol MDC lustre-MDT0000_UUID on MDTidx 0 cmd 800866a1: err = -22
23:06:12:LustreError: 19087:0:(lmv_obd.c:855:lmv_iocontrol()) error: iocontrol MDC lustre-MDT0000_UUID on MDTidx 0 cmd 800866a1: err = -22
...

MDS dmesg log showed

Lustre: DEBUG MARKER: == ost-pools test 23a: OST pools and quota =========================================================== 23:05:40 (1355727940)
Lustre: DEBUG MARKER: lctl pool_new lustre.testpool
Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0006/3]
Lustre: 4861:0:(quota_master.c:793:close_quota_files()) quota[0] is off already
Lustre: 4861:0:(quota_master.c:793:close_quota_files()) Skipped 1 previous similar message
LustreError: 12634:0:(fsfilt-ldiskfs.c:2181:fsfilt_ldiskfs_quotacheck()) quotacheck failed: rc = -22
LustreError: 12634:0:(quota_check.c:112:target_quotacheck_thread()) mdd_obd-lustre-MDT0000: fsfilt_quotacheck: -22

Comments

[Niu Yawei (Inactive)]

This was an interop issue between 2.1 client with 2.4 servers. I think it's not relevant anymore. The last instance on master reported by nasf is a different issue.

[nasf (Inactive)]

Another failure instance on master:
https://testing.hpdd.intel.com/test_sets/314d4b36-d288-11e4-a0e2-5254006e85c2

[Jian Yu]

Lustre client: http://build.whamcloud.com/job/lustre-b2_4/45/ (2.4.1 RC2)
Lustre server: http://build.whamcloud.com/job/lustre-b2_1/215/ (2.1.6)

ost-pools test 23a hit the same failure:
https://maloo.whamcloud.com/test_sets/2f0438ae-1abe-11e3-bf23-52540035b04c

[Sarah Liu]

I reran the test 3 times and cannot reproduce it with server is running 2.1.5 and client is running 2.4

[Sarah Liu]

ok, will let you know when I get the results.

[Niu Yawei (Inactive)]

Hi, Sarah, is it reproducable? Could you try to catch the log on MDS with D_TRACE enabled? So we can see how fsfilt_ldiskfs_quotacheck() failed with -22. Thank you.