[LU-6563] conf-sanity test_53b: FAIL: Assertion 25 failed: (($tstarted >= $tmin2)) (expanded: ((7 >= 8)))

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Blocker
• Reporter: Jian Yu
• Assignee: WC Triage
• Resolution: Fixed
• Labels: dne

Environment

Lustre build: https://build.hpdd.intel.com/job/lustre-master/3009
MDSCOUNT=2

• Severity: 3

Description

conf-sanity test 53b failed as follows under DNE configuration:

CMD: shadow-21vm8 /usr/sbin/lctl set_param mds.MDS.mdt.threads_min=8
mds.MDS.mdt.threads_min=8
CMD: shadow-21vm8 /usr/sbin/lctl set_param mds.MDS.mdt.threads_max=142
mds.MDS.mdt.threads_max=142
CMD: shadow-21vm8 lctl get_param -n mds.MDS.mdt.threads_min
CMD: shadow-21vm8 lctl get_param -n mds.MDS.mdt.threads_max
checking (($tmin2 == ($tmin + $nthrs))) (((8 == (6 + 2))))...
checking (($tmax2 == ($tmax - $nthrs))) (((142 == (144 - 2))))...
CMD: shadow-21vm8 lctl get_param -n mds.MDS.mdt.threads_started
checking (($tstarted >= $tmin2)) (((7 >= 8)))...
 conf-sanity test_53b: @@@@@@ FAIL: Assertion 25 failed: (($tstarted >= $tmin2)) (expanded: ((7 >= 8)))

Maloo report: https://testing.hpdd.intel.com/test_sets/2a054c68-f25c-11e4-9f61-5254006e85c2

Issue Links

• is related to: LU-6206 conf-sanity test_53a: Insane OST thre...

Comments

[Jian Yu]

The failure has started occurring consistently on master branch since 2015-05-02:
https://testing.hpdd.intel.com/sub_tests/query?utf8=%E2%9C%93&test_set[test_set_script_id]=7f66aa20-3db2-11e0-80c0-52540025f9af&sub_test[sub_test_script_id]=286c0182-40a7-11e0-8bad-52540025f9af&sub_test[status]=FAIL&sub_test[query_bugs]=&test_session[test_host]=&test_session[test_group]=&test_session[user_id]=&test_session[query_date]=&test_session[query_recent_period]=&test_node[os_type_id]=&test_node[distribution_type_id]=&test_node[architecture_type_id]=&test_node[file_system_type_id]=&test_node[lustre_branch_id]=24a6947e-04a9-11e1-bb5f-52540025f9af&test_node_network[network_type_id]=&commit=Update+results

[Andreas Dilger]

Hi Yu Jian, please don't use URL shortening services like tinyurl.com. That link may not stick around forever, and it isn't possible to see what it is actually linking to without following the link. Please just use the full URL.

[Andreas Dilger]

Looks like the patch http://review.whamcloud.com/13823 is causing test failures. I will submit a reversion patch.

[Andreas Dilger]

Patch to revert the change:
http://review.whamcloud.com/14682

[James A Simmons]

Ugh, conf-sanity 53X failing again. Such touchy code.

[Andreas Dilger]

I think the problem is that the test is increasing threads_min but it doesn't necessarily do anything to trigger the threads to start. The test probably needs to do something like "touch" or similar before sleeping to ensure the service thread is triggered and will check the ptlrpc_threads_enough() condition. The service thread probably handles some RPCs naturally via ping or DLM lock callback similar some of the time, but not consistently, which is why it is failing intermittently.

[Andreas Dilger]

Have landed patch to revert this.