Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-386] Test failure on test suite replay-single https://jira.whamcloud.com/browse/LU-386 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/27ae3e58-8d47-11e0-aab9-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/27ae3e58-8d47-11e0-aab9-52540025f9af</a>.</p> LU-386

Test failure on test suite replay-single

Bug Blocker Resolved Fixed Zhenyu Xu Maloo Thu, 2 Jun 2011 13:26:51 +0000 Thu, 7 Jul 2011 01:25:53 +0000 Thu, 7 Jul 2011 01:25:53 +0000 Lustre 2.1.0 0 2 <p>this one can be reproduced</p> <p>Bobi</p> <p>I understand that you believe that this is related to bz 10821. Could you please elaborate as to why?</p> <p>Thanks</p> <p>Peter</p> <p>due to misunderstanding i think, there someone associated bug 10821 with this error, which makes me want to check it out to see what history this issue has.</p> <p>Sarah,</p> <p>Would you mind patching this debug patch and reproducing the issue, and uploading /tmp/test_65a file which the debug patch would generate as well as MDS debug logs? Thanks.</p> <p>sure, will do it tomorrow.</p> <p>Is the debug.log MDS log? Because I can not find following messages in it </p> <p>(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms<br/> ..<br/> (fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake</p> <p>as it is set in the test script</p> <p> do_facet $SINGLEMDS lctl set_param fail_val=$((${REQ_DELAY} * 1000))<br/> #define OBD_FAIL_PTLRPC_PAUSE_REQ 0x50a<br/> do_facet $SINGLEMDS $LCTL set_param fail_loc=0x8000050a</p> <p>it is the debug log of MDS and OST, I can separate MDS and OST and test it again </p> <p>better got -1 logs from MDS site if possible (the test_65a file from client site as well)</p> <p>the test_65a is already in the attached.</p> <p>in test_65a-2 i found client got a early reply</p> <p>00000100:00001000:3.0:1308021631.187436:0:9231:0:(events.c:140:reply_in_callback()) @@@ Early reply received: mlen=192 offset=0 replen=216 replied=0 unlinked=0 req@ffff81032c4a9400 x1371560011104309/t0(0) o-1->lustre-MDT0000_UUID@192.168.4.18@o2ib:30/10 lens 232/216 e 0 to 0 dl 1308021637 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1</p> <p>but don't know why client didn't call ptlrpc_at_recv_early_reply()</p> <p>last resort, reproduce the issue and gather -1 logs from client and mds (thus test_65a file also will contain -1 logs from client), you can zip and upload them. </p> <p>from last comment (also reveals in the client log updated in logs.tar.gz), client consumes the early reply on 30/10 protal pair, which are SEQ_METADATA_PORTAL/MDC_REPLY_PORTAL, my local test shows different portal usage (12/10 MDS_REQUEST_PORTAL/MDC_REPLY_PORTAL), I don't know fid seq mechanism much and wondering whether the seq allocation handling relates to the issue.</p> <p>would you mind trying this patch?</p> <p>waiting for build: <a href="http://review.whamcloud.com/#change,957" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,957</a></p> <p>here is the maloo result with your patch<br/> <a href="https://maloo.whamcloud.com/test_sets/5f21a89e-9b79-11e0-9a27-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5f21a89e-9b79-11e0-9a27-52540025f9af</a></p> <p>other logs please find in the attached.</p> <p>still need client test_65 log</p> <p>I've uploaded 65a.tar.gz, please check it.</p> <p>Sarah, </p> <p>I can not reserve any node, so still need your help testing with latest patch set in <a href="http://review.whamcloud.com/#change,957" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,957</a><br/> need -1 mds log and -1 test_65a_client log. (if possible, would it be reproducible to just run this single test case?)</p> <p>TIA</p> <p>yes, it is reproducible for just running test_65a and I will try this patch tomorrow and give you feedback </p> <p>here is the log</p> <p>patch tracking at <a href="http://review.whamcloud.com/1025" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1025</a></p> <p>this patch works. <a href="https://maloo.whamcloud.com/test_sets/9232fa84-a2dc-11e0-aee5-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9232fa84-a2dc-11e0-aee5-52540025f9af</a></p> <p><a href="https://jira.whamcloud.com/browse/LU-320" title="replay-single test_65a: @@@@@@ FAIL: No early reply" class="issue-link" data-issue-key="LU-320"><del>LU-320</del></a> also deals with this issue. The difference is my patch moves away from sysctl. I recommend we move away from sysctl to using $LCTL set_param debug="other". Both test 65a and 65b has this issue.</p> <p>update patch per James suggestion.</p> <p>verified. <a href="https://maloo.whamcloud.com/test_sets/341a370c-a84d-11e0-bd2a-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/341a370c-a84d-11e0-bd2a-52540025f9af</a></p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/194/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #194</a><br/> <a href="https://jira.whamcloud.com/browse/LU-386" title="Test failure on test suite replay-single" class="issue-link" data-issue-key="LU-386"><del>LU-386</del></a> fix replay_single test_65a</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0" class="external-link" target="_blank" rel="nofollow noopener">e0ee0aacd358893dad5c9f0da0dc19ba3ddf08a0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> </ul> <p>landed on master for 2.1.0</p> Development Rank 1|hzv8vb: Rank (Obsolete) 4971 Severity