Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-492] Test failure on sanity-quota test_29 https://jira.whamcloud.com/browse/LU-492 Lustre <p>This issue was created by maloo for bobijam <bobijam@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/ba4bfe42-a856-11e0-bd2a-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ba4bfe42-a856-11e0-bd2a-52540025f9af</a>.</p> LU-492

Test failure on sanity-quota test_29

Bug Blocker Resolved Fixed Zhenyu Xu Maloo Thu, 7 Jul 2011 02:37:09 +0000 Mon, 23 Apr 2012 21:31:12 +0000 Fri, 8 Jul 2011 14:07:36 +0000 Lustre 2.1.0 0 4 <p>the test set client at_max to 10 seconds and sleeps 20 seconds expecting quotactl rpc timeout and exit lfs setquota command.</p> <p>but from client log</p> <blockquote> <p>21:41:13:Lustre: 15526:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1373648715776149 sent from lustre-MDT0000-mdc-ffff81006251e400 to NID 10.10.4.72@tcp has timed out for slow reply: <span class="error">[sent 1310013628]</span> <span class="error">[real_sent 1310013628]</span> <span class="error">[current 1310013672]</span> <span class="error">[deadline 44s]</span> <span class="error">[delay 0s]</span> req@ffff810049e06c00 x1373648715776149/t0(0) o-1->lustre-MDT0000_UUID@10.10.4.72@tcp:12/10 lens 304/304 e 0 to 1 dl 1310013672 ref 2 fl Rpc:XN/ffffffff/ffffffff rc 0/-1</p></blockquote> <p>which shows the rpc is timed out for 44 seconds. This causes the failure.</p> <p>client set request deadline in ptl_send_rpc()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* We give the server rq_timeout secs to process the req, and add the network latency for our local timeout. */ request->rq_deadline = request->rq_sent + request->rq_timeout + ptlrpc_at_get_net_latency(request); </pre> </div></div> <p>where rq_timeout is determined by server's request process time, in ptlrpc_at_set_req_timeout()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> at = &req->rq_import->imp_at; idx = import_at_get_index(req->rq_import, req->rq_request_portal); serv_est = at_get(&at->iat_service_estimate[idx]); req->rq_timeout = at_est2timeout(serv_est); </pre> </div></div> <p>and at_est2timeout()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* add an arbitrary minimum: 125% +5 sec */ return (val + (val >> 2) + 5); </pre> </div></div> <p>so the test should sleep (10*1.25+5) + 10 ~= 28 seconds</p> <p>Seems the test is not we expected, the script set fail_loc to OBD_FAIL_MDS_QUOTACTL_NET, but I didn't find where we check the OBD_FAIL_MDS_QUOTACTL_NET in lustre code, it might be removed by someone mistakenly?</p> <p>patch tracking at <a href="http://review.whamcloud.com/1069" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1069</a></p> <p>niu,</p> <p>please check out mdt_mds_ops[] and in it we defines</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DEF_MDT_HNDL_F(0, QUOTACTL, mdt_quotactl_handle) </pre> </div></div> <p>it will be expanded to </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DEF_HNDL(MDS, GETATTR, _NET, 0, QUOTACTL, mdt_quotactl_handle, &RQF_MDS_QUOTACTL)</pre> </div></div> <p>and further expanded to </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[MDS_QUOTACTL - MDS_QUOTACTL_GETATTR] = { \ .mh_name = QUOTACTL, \ .mh_fail_id = OBD_FAIL_MDS_QUOTACTL_NET, \ .mh_opc = MDS_QUOTACTL, \ .mh_flags = 0, \ .mh_act = mdt_quotactl_handle, \ .mh_fmt = &RQF_MDS_QUOTACTL \ } </pre> </div></div> <p>and in mdt_req_handle(), the fail_loc is checked as follows</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (OBD_FAIL_CHECK_ORSET(h->mh_fail_id, OBD_FAIL_ONCE)) RETURN(0); </pre> </div></div> <p>ah, I see, thank you.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Patch landed for 2.1</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/197/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #197</a><br/> <a href="https://jira.whamcloud.com/browse/LU-492" title="Test failure on sanity-quota test_29" class="issue-link" data-issue-key="LU-492"><del>LU-492</del></a> fix sanity-quota test_29</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2" class="external-link" target="_blank" rel="nofollow noopener">ef803602bf2ee9ed6aabb09aafe23fe036e7b8b2</a><br/> Files : </p> <ul> <li>lustre/tests/sanity-quota.sh</li> </ul> <p>I am running 2.1.1<br/> (see <a href="https://github.com/jlan/lustre-nas/tree/nas-2.1.1" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas/tree/nas-2.1.1</a>)<br/> at both server and client ends. It contains the fix of this ticket.</p> <p>However, it consistenly failed in my 2.1.1 server + 2.1.1 client test environment.</p> <p>test_29()<br/> {<br/> ...</p> <ol> <li>actually send a RPC to make service at_current confined within at_max<br/> $LFS setquota -u $TSTUSR -b 0 -B $BLK_LIMIT -i 0 -I 0 $DIR || error "should succeed"<br/> <=== succeeded</li> </ol> <p>#define OBD_FAIL_MDS_QUOTACTL_NET 0x12e<br/> lustre_fail mds 0x12e<br/> <==== fine</p> <p>$LFS setquota -u $TSTUSR -b 0 -B $BLK_LIMIT -i 0 -I 0 $DIR & pid=$!<br/> <==== "setquota failed: Transport endpoint is not connected"</p> <p>echo "sleeping for 10 * 1.25 + 5 + 10 seconds"<br/> sleep 28<br/> ps -p $pid && error "lfs hadn't finished by timeout"<br/> <==== the process still alive. Die later due to timeout.<br/> ...</p> <p>Is "setquota failed: Transport endpoint is not connected" error expected?<br/> Was that the result of "lustre_fail mds 0x12e"?</p> <p>I tried a "sleep 40" (instead of "sleep 28" after that, and the lfs <br/> command timed out before the check and the test passed. It seems<br/> the sleep formula "10 * 1.25 + 5 + 10 seconds" is not long enough?</p> Development Rank 1|hzv8un: Rank (Obsolete) 4968 Severity