Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-924] Test failure on test suite recovery-small, subtest test_105 https://jira.whamcloud.com/browse/LU-924 Lustre <p>This issue was created by maloo for Chris Gearing <chris@whamcloud.com></p> <p>T</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/467f4a76-25fd-11e1-ae7f-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/467f4a76-25fd-11e1-ae7f-5254004bbbd3</a>.</p> <p>The sub-test test_105 failed with the following error:</p> <blockquote> <p>post-failover df: 1</p></blockquote> <p>Info required for matching: recovery-small 105</p> LU-924

Test failure on test suite recovery-small, subtest test_105

Bug Minor Resolved Fixed Niu Yawei Maloo Wed, 14 Dec 2011 15:40:08 +0000 Fri, 29 Jun 2012 00:48:51 +0000 Wed, 4 Jan 2012 11:13:08 +0000 Lustre 2.2.0 0 1 <p>Jinshan in looking into this one?</p> <p>The problem is clear: the client data was not written into persistent storage before failing over OSTs.</p> <p>From the log, client UUID 8f9b83bd-22e0-c555-20a1-fe22e2547e58 was the old one, and in the test case, we've remounted it new UUID 7419f6fe-3688-8bc3-9560-7367776851cd. However, it didn't update storage in time so when OST was up, it still waited for old client.</p> <p>Hi Jinshan,</p> <p><a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> references a problem which I could read as being because I've used logical volumes for autotest.</p> <p>Can you elaborate please, autotest very much relies on lvm so I'm somewhat concerned.<br/> <span class="error">[19:25:35]</span> Jinshan Xiong: Not entirely, there were a lot of failure about that. the root cause is that we used to wait for transaction to be finsished before returning to client<br/> <span class="error">[19:25:55]</span> Jinshan Xiong: it worked well with hard drives<br/> <span class="error">[19:26:23]</span> Jinshan Xiong: but for lvm, it seems to take longer for data to be flushed<br/> <span class="error">[19:26:23]</span> Chris Gearing: A lot of failures about what, you'll have to forgive me I have no historical knowledge I so need it spelled out!<br/> <span class="error">[19:26:41 | Edited 19:27:01]</span> Chris Gearing: But don't we handshake the data onto the physical media?<br/> <span class="error">[19:26:48]</span> Jinshan Xiong: I don't remember the ticket number<br/> <span class="error">[19:26:58]</span> Jinshan Xiong: but Niu had some fixes about it<br/> <span class="error">[19:27:09]</span> Jinshan Xiong: I will check with him later<br/> <span class="error">[19:27:19]</span> Chris Gearing: Should this be allocated to Niu?<br/> <span class="error">[19:27:22]</span> Jinshan Xiong: On 12/15/11, at 11:26 AM, Chris Gearing wrote:<br/> > But don't we handshake the data onto the physical media?<br/> unfortunately<br/> <span class="error">[19:27:23]</span> Jinshan Xiong: no<br/> <span class="error">[19:27:54]</span> Jinshan Xiong: On 12/15/11, at 11:27 AM, Chris Gearing wrote:<br/> > Should this be allocated to Niu?<br/> Sure, this is the same kind of issue.<br/> <span class="error">[19:28:48]</span> Jinshan Xiong: hmm I shouldn't have faulted lvm<br/> <span class="error">[19:29:00]</span> Jinshan Xiong: this is definitely a lustre issue<br/> <span class="error">[19:29:33]</span> Jinshan Xiong: we should write critical data with OSYNC mode</p> <blockquote> <p>The problem is clear: the client data was not written into persistent storage before failing over OSTs.</p> <p>From the log, client UUID 8f9b83bd-22e0-c555-20a1-fe22e2547e58 was the old one, and in the test case, we've remounted it new UUID 7419f6fe-3688-8bc3-9560-7367776851cd. However, it didn't update storage in time so when OST was up, it still waited for old client.</p></blockquote> <p>We always sync write client data to disk, so I don't see why it wasn't updated.</p> <p>The test failed because the client (which invoke test) was evicted, this client isn't the remounted one, am I right?</p> <p>I don't know the answer to the question about which client was evicted and remounted I'm affraid.</p> <p>Perhaps Jinshan has more knowledge of the test.</p> <p>After looking closer code, I realized that we didn't sync write client data to disk, but just set exp_need_sync = 1 to notify following operation to sync the client data, which could probably is to avoid sync write flood when there are many many clients connecting.</p> <p>And the log shows that the df client (which invoking the test) has been evicted by ost-00000, since ost-0000 didn't find old export in disk and it regards df client as a new client, then the df was not resend after recovery, and result in test failuer at the end.</p> <p>I think we'd better make sure that the client data of df client is synced to disk (otherwise, it could be evicted as a new client during recovery), so there are probably two ways to achieve it:</p> <ul class="alternate" type="square"> <li>sync all osts on ost1 before 'fail ost1';</li> <li>create at least one objects on each ost of ost1 before 'fail ost1';</li> </ul> <p>Jinshan, what's your opinion?</p> <p>patch for master: <a href="http://review.whamcloud.com/1888" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1888</a></p> <p>Though we have talked this over skype, I write it here as a record.</p> <p>I think sync seems fine to me.</p> <p>Sorry for delay response. </p> <p>Landed for 2.2</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/402/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #402</a><br/> <a href="https://jira.whamcloud.com/browse/LU-924" title="Test failure on test suite recovery-small, subtest test_105" class="issue-link" data-issue-key="LU-924"><del>LU-924</del></a> test: sync client data before reboot server (Revision 17a69cf25ed0991e04d85c259f4294dc59734e1e)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=17a69cf25ed0991e04d85c259f4294dc59734e1e" class="external-link" target="_blank" rel="nofollow noopener">17a69cf25ed0991e04d85c259f4294dc59734e1e</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> Related LU-1582 Development Rank 1|hzv7qn: Rank (Obsolete) 4788 Severity