https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-482 Lustre <p>This issue was created by maloo for yujian &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/38eaf1c4-a6f2-11e0-bd2a-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/38eaf1c4-a6f2-11e0-bd2a-52540025f9af</a>.</p> <p>The sub-test test_0a failed with the following error:</p> <blockquote> <p>Restart of mds1 failed!</p></blockquote> <hr /> <p>Info required for matching: replay-dual 0a</p> LU-482

Test failure on test suite replay-dual, subtest test_0a

Bug Minor Resolved Cannot Reproduce WC Triage Maloo Tue, 5 Jul 2011 08:46:55 +0000 Tue, 7 Jun 2016 22:09:14 +0000 Mon, 2 Feb 2015 19:24:14 +0000 Lustre 2.3.0 Lustre 2.1.3 Lustre 1.8.8 0 16 <p>Console log on the MDS node showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>21:07:08:LDISKFS-fs (dm-0): ldiskfs_check_descriptors: Checksum for group 128 failed (37201!=47318) 21:07:08:LDISKFS-fs (dm-0): group descriptors corrupted! 21:07:08:LustreError: 24509:0:(obd_mount.c:1392:server_kernel_mount()) ll_kern_mount failed: rc = -22 21:07:08:LustreError: 24509:0:(obd_mount.c:1672:server_fill_super()) Unable to mount device /dev/mapper/lvm-P0: -22 21:07:08:LustreError: 24509:0:(obd_mount.c:2151:lustre_fill_super()) Unable to mount (-22) 21:07:08:Lustre: DEBUG MARKER: replay-dual test_0a: @@@@@@ FAIL: Restart of mds1 failed! </pre> </div></div> <p>Niu will look into this one</p> <p>I recalled that there are several similar metadata corruption reports on lustre-discuss, seems they were all running MDT on dm device. I suspect it's kind of issue of running ext4 on dm device, but I didn't find any bug report on ext4 maillist yet.</p> <p>just found several other different symptoms:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">04:38:26:LustreError: 14663:0:(md_local_object.c:433:llo_local_objects_setup()) creating obj [REM_OBJ_DIR] fid = [0x200000001:0xc:0x0] rc = -116 04:38:26:LustreError: 14663:0:(mdt_handler.c:4603:mdt_init0()) Can't init device stack, rc -116 04:38:26:LustreError: 14663:0:(obd_config.c:519:class_setup()) setup lustre-MDT0000 failed (-116) 04:38:26:LustreError: 14663:0:(obd_config.c:1358:class_config_llog_handler()) Err -116 on cfg command: 04:38:26:Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f 04:38:26:LustreError: 15c-8: MGC10.10.4.72@tcp: The configuration from log <span class="code-quote">'lustre-MDT0000'</span> failed (-116). This may be the result of communication errors between <span class="code-keyword">this</span> node and the MGS, a bad configuration, or other errors. See the syslog <span class="code-keyword">for</span> more information. 04:38:27:LustreError: 14624:0:(obd_mount.c:1183:server_start_targets()) failed to start server lustre-MDT0000: -116 04:38:27:LustreError: 14624:0:(obd_mount.c:1710:server_fill_super()) Unable to start targets: -116 04:38:27:LustreError: 14624:0:(obd_config.c:564:class_cleanup()) Device 3 not setup 04:38:32:LustreError: 14624:0:(obd_mount.c:2151:lustre_fill_super()) Unable to mount (-116) </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">03:41:33:LustreError: 11951:0:(obd_mount.c:291:ldd_parse()) cannot open CONFIGS/mountdata: rc = -2 03:41:34:LustreError: 11951:0:(obd_mount.c:1364:server_kernel_mount()) premount parse options failed: rc = -2 03:41:34:LustreError: 11951:0:(obd_mount.c:1672:server_fill_super()) Unable to mount device /dev/mapper/lvm-P0: -2 03:41:34:LustreError: 11951:0:(obd_mount.c:2151:lustre_fill_super()) Unable to mount (-2) </pre> </div></div> <p>The failure can also be found in early test which running over non-lvm device: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">06:30:34:LustreError: 24808:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** 06:30:34:Turning device sdb (0x800010) read-only 06:30:34:Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 06:30:35:Lustre: Failing over lustre-MDT0000 06:30:35:Lustre: Skipped 9 previous similar messages 06:30:35:Lustre: mdd_obd-lustre-MDT0000-0: shutting down <span class="code-keyword">for</span> failover; client state will be preserved. 06:30:35:LustreError: 24914:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 06:30:35:LustreError: 24914:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 3 previous similar messages 06:30:35:LustreError: 24914:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 06:30:35:LustreError: 24914:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 3 previous similar messages 06:30:41:Removing read-only on unknown block (0x800010) 06:30:41:Lustre: server umount lustre-MDT0000 complete 06:30:41:Lustre: Skipped 3 previous similar messages 06:30:52:LDISKFS-fs (sdb): ldiskfs_check_descriptors: Checksum <span class="code-keyword">for</span> group 128 failed (9760!=628) 06:30:52:LDISKFS-fs (sdb): group descriptors corrupted! 06:30:52:LustreError: 25027:0:(obd_mount.c:1394:server_kernel_mount()) ll_kern_mount failed: rc = -22 06:30:52:LustreError: 25027:0:(obd_mount.c:1675:server_fill_super()) Unable to mount device /dev/sdb: -22 06:30:52:LustreError: 25027:0:(obd_mount.c:2149:lustre_fill_super()) Unable to mount (-22) 06:30:52:Lustre: DEBUG MARKER: replay-dual test_0a: @@@@@@ FAIL: Restart of mds1 failed! </pre> </div></div> <p>Hi, Chris</p> <p>Looks this failure often happen on vm, could you reserve a vm node for me to do the manual test? (seems now all vm node are reserved for auto-test). Thanks. </p> <p>I'm creating a VM, a little bit delay because mjmac is working on git issues. As soon as it's up I will let you know. It will be client-14vm<span class="error">&#91;1-4&#93;</span>.</p> <p>Everything like loadjenkinsbuild should work on it.</p> <p>As I said, I'll update when it's ready.</p> <p>Niu, can I provide any more assistance to you one this. It’s not clear to me how I can help but if there is a way please ask.</p> <p>Hi, Chris<br/> I didn't realize that the client14vm<span class="error">&#91;1-4&#93;</span> are already ready, I'm going to reproduce the bug on these vm nodes. Since the client-14vm<span class="error">&#91;1-4&#93;</span> are reserved by you, I can't install lustre-master on it (seems 1.8.6 is installed on them), could you help me to install lustre-master (both server and client) on them or let me reserve these vm nodes? Thank you in advance.</p> <p>Hi,</p> <p>How long will you require it for? The problem is that client-14 is the <br/> only system I have for development, I'm running a 1.8.6 client vs 2.1 <br/> test at the moment. Once this is done I'll release it and register it in <br/> your name, but I will need it back as soon as possible.</p> <p>Chris</p> <p>OK, I've finished with the nodes, they are now reserved in your name.</p> <p>Thank you, Chris. I think two days is enough for me, so you can get it back next week.</p> <p>Could you show me how to create the lvm device for testing? Looks the auto-test vm nodes usually uses /dev/vda4 as the physical volume, but I didn't find /dev/vda4 on the client-14 vm nodes. </p> <p>vm1 and vm2 are clients, vm3 is the MDS, vm4 is the OSS</p> <p>So I partition sda4 on each of the server nodes to be the rest of the disk.</p> <p>fdisk /dev/vda &lt; Create4thPartition</p> <p><span class="error">&#91;I&#39;ve attached Create4thPartition&#93;</span></p> <p>Then I reboot the node.</p> <p>Then on the OSS.</p> <p>vgcreate lvm /dev/vda4<br/> lvcreate -L11GB -nP0 lvm <br/> lvcreate -L11GB -nP1 lvm <br/> lvcreate -L11GB -nP2 lvm <br/> lvcreate -L11GB -nP3 lvm <br/> lvcreate -L11GB -nP4 lvm <br/> lvcreate -L11GB -nP5 lvm <br/> lvcreate -L11GB -nP6 lvm <br/> lvcreate -L11GB -nP7 lvm <br/> lvcreate -L11GB -nP8 lvm <br/> lvcreate -L11GB -nP9 lvm <br/> vgchange -a y lvm</p> <p>On the mds</p> <p>vgcreate lvm /dev/vda4<br/> lvcreate -L26GB -nP0 lvm <br/> vgchange -a y lvm</p> <p>Then you have the disks.</p> <p>Thanks a lot, Chris. I tried ran replay-dual 0a and replay-ost-single + replay-dual 0a many times on the lvm device, but failed to reproduce the bug.</p> <p>I don't see why replay-single 0a doesn't suffer from this failure, there is an extra 'sleep 10' in the replay-single 0a, I'm thinking that we could also add 'sleep 10' to the replay-dual 0a, then enable it in the auto-test to see if can be reproduced? Peter, Chris, is it ok? Thanks.</p> <p>Oleg would be ok with this as a test but would not want to make a habit of this practice</p> <p>Thank you, Peter. My gerrit account has some problem recently, so I can't push patch for review now, I've asked Robert for help, and will push this diagnostic patch as soon as the account get ready.</p> <p>This sounds OK you should be careful to fully comment the push so that <br/> everybody realizes this is not for review and only for test, I can then <br/> repeatedly test it in autotest.</p> <p>Chris</p> <p>Dropping as blocker for now. May re-add if it starts reoccurring regularly.</p> <p>Ok, I pushed the patch at <a href="http://review.whamcloud.com/#change,1157" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1157</a>, let's see if the bug is reproduceable after the patch is pushed. Thanks.</p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/048f8c56-bdfd-11e0-8bdf-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/048f8c56-bdfd-11e0-8bdf-52540025f9af</a></p> <p>Another failure <a href="https://maloo.whamcloud.com/test_sets/b965245c-e089-11e0-9909-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b965245c-e089-11e0-9909-52540025f9af</a></p> <p>another failure: <a href="https://maloo.whamcloud.com/test_sets/fac8e9cc-e5bd-11e0-9909-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fac8e9cc-e5bd-11e0-9909-52540025f9af</a></p> <p>another failure: <a href="https://maloo.whamcloud.com/test_sets/669543a6-e62e-11e0-9909-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/669543a6-e62e-11e0-9909-52540025f9af</a></p> <p>another failure instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/2db5105e-ec85-11e0-9909-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2db5105e-ec85-11e0-9909-52540025f9af</a></p> <p>another failure:</p> <p><a href="https://maloo.whamcloud.com/test_sets/3c4a840a-f5ef-11e0-9b90-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3c4a840a-f5ef-11e0-9b90-52540025f9af</a></p> <p>another failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/08615aa2-f6a7-11e0-a451-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/08615aa2-f6a7-11e0-a451-52540025f9af</a></p> <p>I'm wondering if it's already been fixed in <a href="https://jira.whamcloud.com/browse/LU-699" title="replay-dual test_1 fails to remount mdt" class="issue-link" data-issue-key="LU-699"><del>LU-699</del></a>. Is there any failure after 886e67d4fe87d293952a11e7f41b98a8c3abeddd ?</p> <p>Lai will look into this one</p> <p>A failure after <a href="https://jira.whamcloud.com/browse/LU-699" title="replay-dual test_1 fails to remount mdt" class="issue-link" data-issue-key="LU-699"><del>LU-699</del></a> landed: <a href="https://maloo.whamcloud.com/test_sets/4fed926e-0313-11e1-bb4f-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4fed926e-0313-11e1-bb4f-52540025f9af</a></p> <p>In MDS dmesg, it shows barrier is not supported and disabled:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LDISKFS-fs (dm-0): mounted filesystem with ordered data mode JBD: barrier-based sync failed on dm-0-8 - disabling barriers </pre> </div></div> <p>There is an <a href="http://serverfault.com/questions/279571/lvm-dangers-and-caveats" class="external-link" target="_blank" rel="nofollow noopener">article about barrier support on LVM and on VM hypervisor</a>: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Write caching and write re-ordering by the hard drive is important to good performance, but due to lack of complete write barrier support in kernel versions &lt; 2.6.33 (2.6.31 has some support, while 2.6.33 works for all types of device target), you may well find that a large amount of FS metadata (including journals) is lost by a hard crash that leaves data in the hard drives' write buffers. ... Running LVM on top of a VM hypervisor such as VMware, Xen or KVM can create similar problems to a kernel without write barriers, due to write caching and write re-ordering. </pre> </div></div> <p>So if kernel &lt; 2.6.33, barrier is not fully supported by LVM, and if system is deployed upon VM hypervisor, even kernel is okay, the similar data lost will be seen.</p> <p>I'm afraid we can't really fix this issue on current test environment, if Niu's patch (sleep 10 seconds before test start) can circumvent this failure, it's an acceptable solution.</p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/04174d90-071d-11e1-b0d9-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/04174d90-071d-11e1-b0d9-52540025f9af</a></p> <blockquote> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/04174d90-071d-11e1-b0d9-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/04174d90-071d-11e1-b0d9-52540025f9af</a></p></blockquote> <p>This failure looks different, not sure if they are same.</p> <p>It maybe not the same as the original issue, but it is quite similar as the failure case you mentioned at 11/Jul/11 10:16 AM:</p> <p>09:34:54:LustreError: 3764:0:(md_local_object.c:433:llo_local_objects_setup()) creating obj <span class="error">&#91;fld&#93;</span> fid = <span class="error">&#91;0x200000001:0x3:0x0&#93;</span> rc = -116<br/> 09:34:54:LustreError: 3764:0:(mdt_handler.c:4571:mdt_init0()) Can't init device stack, rc -116<br/> 09:34:54:LustreError: 3764:0:(obd_config.c:522:class_setup()) setup lustre-MDT0000 failed (-116)<br/> 09:34:54:LustreError: 3764:0:(obd_config.c:1361:class_config_llog_handler()) Err -116 on cfg command:<br/> 09:34:54:Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f <br/> 09:34:54:LustreError: 15c-8: MGC10.10.4.114@tcp: The configuration from log 'lustre-MDT0000' failed (-116). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> 09:34:54:LustreError: 3725:0:(obd_mount.c:1271:server_start_targets()) failed to start server lustre-MDT0000: -116<br/> 09:34:55:LustreError: 3725:0:(obd_mount.c:1803:server_fill_super()) Unable to start targets: -116<br/> 09:34:55:LustreError: 3725:0:(obd_config.c:567:class_cleanup()) Device 3 not setup<br/> 09:34:59:LustreError: 3725:0:(obd_mount.c:2306:lustre_fill_super()) Unable to mount (-116)<br/> 09:35:00:Lustre: DEBUG MARKER: replay-dual test_0a: @@@@@@ FAIL: Restart of mds1 failed!</p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/edb71d5e-0803-11e1-b0d9-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/edb71d5e-0803-11e1-b0d9-52540025f9af</a></p> <p>I reviewed all of this again and I am not suspecting the no barrier has little to do with our problems.</p> <p>The barrier is important for hard crashes of the nodes which does not happen in our tests most of the time. Instead we do readonly trick where all subsequent writes to devices would fail. Of course first we do sync to flush currently dirty data.</p> <p>My suspicion is a single sync would only move data from dirty cache of lvm to dirty cache of next in line block device and depending on how the readonly was implemented it still might be discarded in the end.</p> <p>So I have an alternative patch here: <a href="http://review.whamcloud.com/1656" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1656</a> that would hopefully work better.</p> <p>I was tempted to do "sync ; sleep 1 ; sync", but that would add an extra 1 second delay to all failover testing which is probably way too much overhead.</p> <p>Also note that this bug might hit tests other than 1st one after all where some of important data we expect to be committed would not be committed just as journal or initial formatting data would be lost in the first test.</p> <p>In fact, this failure is also seen on non-lvm device in real machine, see my previous comment on 12/Jul/11 12:53 AM, or check the link:<a href="https://maloo.whamcloud.com/test_sets/e69561f8-82c4-11e0-b4df-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e69561f8-82c4-11e0-b4df-52540025f9af</a></p> <p>So I still don't quite understand why it could happen.</p> <p>From the failure log, we can see when mds restart after replay_barrier(), the fisrt kernel mount always success, then it failed for:</p> <ul class="alternate" type="square"> <li>can't find config file, or;</li> <li>checksum for group failure on next mount with the mount option in config file, or;</li> <li>fail to find local objects or llog file after the second mount (it's rare compare with the above two failures);</li> </ul> <p>so my guess is that the filesystem is corrupt after first mount, which might be caused by journal recovery during first mount?</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/344/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #344</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test-framework: Ensure dirty cache is flushed before barrier (Revision a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95" class="external-link" target="_blank" rel="nofollow noopener">a0223bfc05d4f3d5aa3ee88229f6924fe0e68c95</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> </ul> <p>the failure is still showing up: <a href="https://maloo.whamcloud.com/test_sets/87c26e80-0abb-11e1-b0d9-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/87c26e80-0abb-11e1-b0d9-52540025f9af</a></p> <p>In the replay_barrier(), we did a create after sync but before set rdonly, I think that could cause trouble if the set rdonly happened in the middle of journal commit, though it can't explain why we never seen the failure on the following tests or replay-single, but anyway, I think we'd better move this create before sync.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/366/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #366</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> replay-dual test_0a failed (Revision b1bec2c47aa597cf96e60840b8dd97125d782a03)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=b1bec2c47aa597cf96e60840b8dd97125d782a03" class="external-link" target="_blank" rel="nofollow noopener">b1bec2c47aa597cf96e60840b8dd97125d782a03</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Please see comment in <a href="http://review.whamcloud.com/1157" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1157</a>. I'd like to make this timeout conditional on the kernel version instead of permanent.</p> <p>This is being addressed in <a href="http://review.whamcloud.com/#change,1761" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1761</a></p> <p>also hit in replay-ost-single test (<a href="https://maloo.whamcloud.com/test_sets/08bf83f4-2a8f-11e1-a412-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/08bf83f4-2a8f-11e1-a412-5254004bbbd3</a>)</p> <p>from OSS console log of test 0</p> <p>22:20:49:LDISKFS-fs (dm-5): recovery complete<br/> 22:20:49:LDISKFS-fs (dm-5): mounted filesystem with ordered data mode<br/> 22:20:50:LDISKFS-fs (dm-5): ldiskfs_check_descriptors: Checksum for group 128 failed (9336!=38256)<br/> 22:20:50:LDISKFS-fs (dm-5): group descriptors corrupted!<br/> 22:20:50:LustreError: 2384:0:(obd_mount.c:1485:server_kernel_mount()) ll_kern_mount failed: rc = -22<br/> 22:20:50:LustreError: 2384:0:(obd_mount.c:1765:server_fill_super()) Unable to mount device /dev/mapper/lvm--OSS-P5: -22<br/> 22:20:50:LustreError: 2384:0:(obd_mount.c:2306:lustre_fill_super()) Unable to mount (-22)<br/> 22:20:50:Lustre: DEBUG MARKER: replay-ost-single test_0b: @@@@@@ FAIL: Restart of ost6 failed!</p> <p>Hit this three times in a row for a patch that cannot possibly be causing problems.</p> <p><a href="https://maloo.whamcloud.com/test_sessions/55893158-58f2-11e1-a226-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/55893158-58f2-11e1-a226-5254004bbbd3</a> (RHEL5.7 x86_64)<br/> <a href="https://maloo.whamcloud.com/test_sessions/e635a72c-5d93-11e1-b5d6-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/e635a72c-5d93-11e1-b5d6-5254004bbbd3</a> (RHEL6.2 x86_64)<br/> <a href="https://maloo.whamcloud.com/test_sessions/acca96e0-5e25-11e1-8401-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/acca96e0-5e25-11e1-8401-5254004bbbd3</a> (RHEL6.2 x86_64)</p> <p>I wonder if there is some other way to avoid this problem? I notice that while replay-dual.sh has a "sleep 10" after formatting the filesystem, it does not have "sync" before that, so it may well be that the data is not even starting to be flushed to the underlying disk yet. Once test_0a() sets replay_barrier it may discard all of the dirty data on the device.</p> <p>Adding a sync before the sleep is one simple solution that may hide this problem.</p> <p>Another possibility is to stop using LVM inside the VM guest, and instead map partitions or LVs in the host to be disk devices in the VM so that at least the flush operations in the guest will push the data from the guest out to the host where it will be persistent even if it is not on disk in the host.</p> <p>I've submitted a new test fix in <a href="http://review.whamcloud.com/2223" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2223</a>, which adds sync; wait; sync before the first test is run, to avoid losing the blocks from just-formatted filesystems. This also includes the change I requested in <a href="http://review.whamcloud.com/1761" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1761</a> to limit this workaround to the affected kernels (currently all vendor kernels), so that we don't keep carrying this hack indefinitely in the future.</p> <p>Since this bug only hits in about 10% of test failures, it won't immediately be clear whether this change is resolving the problem, but is itself not harmful to the code and can be landed to master (after a successful autotest) so that it is included in all test runs.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Hit this issue when doing 2.2-RC1 test, RHEL6 server and client:<br/> <a href="https://maloo.whamcloud.com/test_sets/0a633ea6-70e6-11e1-a89e-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0a633ea6-70e6-11e1-a89e-5254004bbbd3</a></p> <p>Similar failure:</p> <p><a href="https://maloo.whamcloud.com/test_sets/33c75cc0-7233-11e1-8ebe-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/33c75cc0-7233-11e1-8ebe-5254004bbbd3</a></p> <p>The failure shows mountdata file is missing on MGS, and previous fix added sync on lustre client, IMO it won't help for MGS. I'll submit a patch to do sync on MGS to see how it goes.</p> <p><a href="http://review.whamcloud.com/#change,2545" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2545</a></p> <p>Landed for 2.3</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=ofa/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,ofa #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=ofa/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,ofa #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=ofa/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,ofa #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=ofa/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,ofa #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/487/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #487</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)<br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)<br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)<br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)<br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)<br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)<br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync new fs before first replay test (Revision 879e8d045057941ae0a5117d096f53975ef12ad0)<br/> <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> test: sync MDS before first replay test (Revision c3d36069ca342a2f1ee97f983e7c49ae540688e5)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=879e8d045057941ae0a5117d096f53975ef12ad0" class="external-link" target="_blank" rel="nofollow noopener">879e8d045057941ae0a5117d096f53975ef12ad0</a><br/> Files : </p> <ul> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/replay-dual.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=c3d36069ca342a2f1ee97f983e7c49ae540688e5" class="external-link" target="_blank" rel="nofollow noopener">c3d36069ca342a2f1ee97f983e7c49ae540688e5</a><br/> Files : </p> <ul> <li>lustre/tests/replay-dual.sh</li> <li>lustre/tests/replay-single-lmv.sh</li> <li>lustre/tests/replay-single.sh</li> </ul> <p>This is still being hit in testing on master.</p> <p>This is still failing about 10% of the time:</p> <p><a href="https://maloo.whamcloud.com/test_sets/697e6e7c-94c9-11e1-a028-525400d2bfa6" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/697e6e7c-94c9-11e1-a028-525400d2bfa6</a><br/> <a href="https://maloo.whamcloud.com/test_sets/a328ce1e-9498-11e1-a028-525400d2bfa6" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a328ce1e-9498-11e1-a028-525400d2bfa6</a><br/> <a href="https://maloo.whamcloud.com/test_sets/9eb614a2-9436-11e1-a028-525400d2bfa6" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9eb614a2-9436-11e1-a028-525400d2bfa6</a><br/> <a href="https://maloo.whamcloud.com/test_sets/66153ebe-9375-11e1-b37e-525400d2bfa6" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/66153ebe-9375-11e1-b37e-525400d2bfa6</a><br/> <a href="https://maloo.whamcloud.com/test_sets/27aea4c4-9372-11e1-b37e-525400d2bfa6" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/27aea4c4-9372-11e1-b37e-525400d2bfa6</a><br/> <a href="https://maloo.whamcloud.com/test_sets/08c3f5c0-9311-11e1-9e8b-525400d2bfa6" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/08c3f5c0-9311-11e1-9e8b-525400d2bfa6</a></p> <p>This is still causing 15% of all test runs to fail. I'm thinking that before we take the drastic step of disabling this test from being run entirely, it makes sense to catch the test_0a failure as it happens, and exit from the script in that case, but continue running the rest of replay-dual if test_0a passes. Something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mount_facets () { local facets=${1:-$(get_facets)} local facet for facet in ${facets//,/ }; do if [ "$TESTSUITE.$TESTNAME" = "replay-dual.test_0a" ]; then mount_facet $facet || skip "Restart of $facet failed! See LU-482." else mount_facet $facet || error "Restart of $facet failed!" fi done } </pre> </div></div> <p>and then in test_0a() it should check whether the MDS was actually restarted after facet_failover returns, or abort the test entirely (exit 0).</p> <p>Also, for posterity, here is the actual test failure output, which doesn't exist anywhere in this bug report.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== replay-dual test 0a: expired recovery with lost client ============================================ 21:49:18 (1336538958) Filesystem 1K-blocks Used Available Use% Mounted on client-23vm3@tcp:/lustre 49867944 2266012 45068324 5% /mnt/lustre total: 50 creates in 0.18 seconds: 285.24 creates/second fail_loc=0x80000514 Failing mds1 on node client-23vm3 Stopping /mnt/mds1 (opts:) on client-23vm3 affected facets: mds1 Failover mds1 to client-23vm3 21:49:37 (1336538977) waiting for client-23vm3 network 900 secs ... 21:49:37 (1336538977) network interface is UP Starting mds1: -o user_xattr,acl /dev/lvm-MDS/P1 /mnt/mds1 client-23vm3: mount.lustre: mount /dev/mapper/lvm--MDS-P1 at /mnt/mds1 failed: No such file or directory client-23vm3: Is the MGS specification correct? client-23vm3: Is the filesystem name correct? client-23vm3: If upgrading, is the copied client log valid? (see upgrade docs) mount -t lustre /dev/lvm-MDS/P1 /mnt/mds1 Start of /dev/lvm-MDS/P1 on mds1 failed 2 replay-dual test_0a: @@@@@@ FAIL: Restart of mds1 failed! </pre> </div></div> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/628fd846-9d7d-11e1-a1d8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/628fd846-9d7d-11e1-a1d8-52540035b04c</a></p> <p>I've uploaded <a href="http://review.whamcloud.com/2731" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2731</a>, which will abort replay-dual if test_0a fails to mount the MDS. This should allow 90% of replay-dual tests to continue to run properly, while avoiding spurious test failures due to this bug.</p> <p>Lustre Tag: v1_8_8_WC1_RC1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/195/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/195/</a><br/> Distro/Arch: RHEL5.8/x86_64(server), RHEL6.2/x86_64(client)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>replay-ost-single test 0b failed with the similar issue:<br/> <a href="https://maloo.whamcloud.com/test_sets/de67db1a-9d66-11e1-8587-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/de67db1a-9d66-11e1-8587-52540035b04c</a></p> <p>Console log on the OSS node (client-30vm4) showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>06:47:59:Lustre: DEBUG MARKER: mkdir -p /mnt/ost1; mount -t lustre /dev/lvm-OSS/P1 /mnt/ost1 06:47:59:LDISKFS-fs (dm-0): ldiskfs_check_descriptors: Checksum for group 0 failed (60782!=34767) 06:48:00:LDISKFS-fs (dm-0): group descriptors corrupted! 06:48:00:LustreError: 12142:0:(obd_mount.c:1307:server_kernel_mount()) premount /dev/lvm-OSS/P1:0x0 ldiskfs failed: -22, ldiskfs2 failed: -19. Is the ldiskfs module available? 06:48:00:LustreError: 12142:0:(obd_mount.c:1633:server_fill_super()) Unable to mount device /dev/lvm-OSS/P1: -22 06:48:00:LustreError: 12142:0:(obd_mount.c:2065:lustre_fill_super()) Unable to mount (-22) 06:48:01:Lustre: DEBUG MARKER: /usr/sbin/lctl mark replay-ost-single test_0b: @@@@@@ FAIL: Restart of ost1 failed! </pre> </div></div> <p>recovery-double-scale also failed with the similar issue:<br/> <a href="https://maloo.whamcloud.com/test_sets/f43de6fe-9c36-11e1-8837-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f43de6fe-9c36-11e1-8837-52540035b04c</a></p> <p>It looks like the first failure with my patch applied is in: <br/> <a href="https://maloo.whamcloud.com/test_sets/64042a50-a242-11e1-bba6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/64042a50-a242-11e1-bba6-52540035b04c</a></p> <p>The test_0a() is reported to pass, but for some reason the test is marked is timed out. I'm not sure why the test script is reported as a failure, since all of the steps to exit the script normally should be run if the LU482 file is detected. </p> <p>Lustre Branch: master<br/> Lustre Build: <a href="http://build.lab.whamcloud.com/job/lustre-master/555" class="external-link" target="_blank" rel="nofollow noopener">http://build.lab.whamcloud.com/job/lustre-master/555</a><br/> Distro/Arch: RHEL5/x86_64(client), RHEL6/x86_64(server)</p> <p>The lustre-initialization-1 stage failed with the same issue:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00:59:22:Lustre: DEBUG MARKER: mkdir -p /mnt/ost13; mount -t lustre /dev/lvm-OSS/P6 /mnt/ost13 00:59:22:LDISKFS-fs (dm-5): recovery complete 00:59:22:LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: 00:59:22:LDISKFS-fs (dm-5): ldiskfs_check_descriptors: Checksum for group 0 failed (16572!=40708) 00:59:22:LDISKFS-fs (dm-5): group descriptors corrupted! 00:59:22:LustreError: 6307:0:(obd_mount.c:1499:server_kernel_mount()) vfs_kern_mount failed: rc = -22 00:59:22:LustreError: 6307:0:(obd_mount.c:1794:server_fill_super()) Unable to mount device /dev/mapper/lvm--OSS-P6: -22 00:59:22:LustreError: 6307:0:(obd_mount.c:2359:lustre_fill_super()) Unable to mount (-22) </pre> </div></div> <p><a href="https://maloo.whamcloud.com/test_sessions/0777c22a-a129-11e1-b20c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/0777c22a-a129-11e1-b20c-52540035b04c</a></p> <p>It seems that the client (I think?) is not being cleaned up properly (needs a "umount -f $DIR"?), and possibly the script needs to do "exit 0" in the error handling case.</p> <p>Keith</p> <p>Could you please look into why this tests sometimes fails in vm environments?</p> <p>Thanks</p> <p>Peter</p> <p>I am working on recreating this issue on my laptop VMs. </p> <p>This error</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00:59:22:LDISKFS-fs (dm-5): group descriptors corrupted! </pre> </div></div> <p>Seems to be a common point that isn't good. </p> <p>Was there ever a time we didn't see this issue? </p> <p>We only see it in VM + LVM/DM + LDISKFS-fs? I tried for a few hours today in a loop to trigger it without LVM/DM on a very basic setup without luck. Tomorrow I will try and emulate the 8 osts + LVM/DM setup. </p> <p>Have we had different failure rates with and without the various patches in Master? </p> <p>Hi Keith, can you please make a patch to disable this test and add it back after this issue has been fixed?</p> <p>I carried out a bunch of VM + Physical testing and found that the error appear possible on either. The results are in these Maloo sessions;</p> <p>Physical:<br/> <a href="https://maloo.whamcloud.com/test_sessions/2249ff5a-a845-11e1-bdf9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/2249ff5a-a845-11e1-bdf9-52540035b04c</a> <span class="error">&#91;Good case full review test and replay-dual test 0 failed&#93;</span><br/> <a href="https://maloo.whamcloud.com/test_sessions/287b6b26-a7ae-11e1-acdf-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/287b6b26-a7ae-11e1-acdf-52540035b04c</a> <span class="error">&#91;Repeated replay-dual, replay-single no failures&#93;</span><br/> <a href="https://maloo.whamcloud.com/test_sessions/33371b22-a700-11e1-acdf-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/33371b22-a700-11e1-acdf-52540035b04c</a> <span class="error">&#91;Repeated replay-dual, replay-single no failures&#93;</span></p> <p>VM:<br/> <a href="https://maloo.whamcloud.com/test_sessions/822ee7e4-a83d-11e1-bdf9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/822ee7e4-a83d-11e1-bdf9-52540035b04c</a> <span class="error">&#91;Passed, full review test&#93;</span><br/> <a href="https://maloo.whamcloud.com/test_sessions/b16db6fe-a70a-11e1-99eb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/b16db6fe-a70a-11e1-99eb-52540035b04c</a> <span class="error">&#91;Repeated replay-dual, replay-single 1 failure&#93;</span><br/> TDB <span class="error">&#91;Repeated replay-dual, XXX failure&#93;</span></p> <p>So I don't think it's in anyway safe to say this is a VM issue, because the first case is physical.</p> <p>Function replay_barrier() runs mcreate after syncing target, this looks suspicious, I'll commit a patch to move sync after mcreate.</p> <p>Patch is at: <a href="http://review.whamcloud.com/#change,2931" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2931</a>, Chris, could you help conduct repeated test against this patch?</p> <p>1500 runs later on my laptop VM setup: No sign of this issue but a hard panic on the MDS. The panic hung the box and I didn't have serial running so only a 90% screen shot of the panic. </p> <p>It was 3 days of testing and I didn't see "this" issue. Since we are seeing alot of failures I am lead to believe our visualization test environment is part of the issue. My laptop run ran the MDS and OSTs on SSD, I am going to move them fully to a single spindle put them into a constrained IO environment. </p> <p>If it is agreed that we want to skip this test I can provided a patch. It seems we have been living with this problem for a while and I don't know the full history. </p> <p>Here are my thoughts of the issue and how to hopefully better reproduce it.</p> <p>It was noted when we moved to IU this was hit much more often.</p> <p>So the theory is when you start with a fresh (zero filled essentially) disk image for would be lustre FS, when the initial lustre format fails to make it to disk, the mdt mount fails.</p> <p>So for reproduction it seems it's a good idea to zero-fill (important for ext4) parts of the underlying disk before every test restart <em>on the host</em>, outside of VM.<br/> Also I guess it's important to have MDS in a separate VM too (at least at the beginning).</p> <p>If tis is confirmed, then the efforts could be 2-way. One is to perhaps try to impose a lustre-looking fs image at the fs (as a temp workaround), and then in parallel seeing how to make the mdt actually write stuff to disk.</p> <p>Well I have been running a few different test paths and I thought I would make notes of them here. </p> <p>1st: Running the test directly in any form I still have not been able to locally trigger the issue. </p> <p>2: I wrote a quick script do to an IO to the LVM device, mkfs.lustre, mount, umount loop. <br/> 2a: Running 7 OST mounts in this loop is not safe. Normally within 1-2 hours one of the devices will fail to mount. Error logs say timeout to the MDS but the device can be mounted 1 second later. <br/> 2b. While running 7 separate MDS/MDT device I run into trouble within a few hours. Generally while trying to mount a device it seems to think it was already mounted. There may be some loose accounting going on with the unmount or ??? I have ran for about 24 hours with a single device / thread. This MDS issue maybe related to the issues reported here and I focusing around the MDS/MDT mount failures first. </p> <p>It not quite clear if this make/mount/unmount loop test is related this this bug but is seems worth investigating as the real test produces similar actions. </p> <p>Latest workaround patch has made this less invasive so dropping priority</p> <p>Reduced to Critical because it is not happening as often.</p> <p>More instances on Lustre 2.1.3 RC1:<br/> <a href="https://maloo.whamcloud.com/test_sets/c0df6464-e48d-11e1-af05-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c0df6464-e48d-11e1-af05-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/465543a4-e3ab-11e1-b6d3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/465543a4-e3ab-11e1-b6d3-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/4b23f22e-e3ea-11e1-b6d3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4b23f22e-e3ea-11e1-b6d3-52540035b04c</a></p> <p>More instances on Lustre 2.1.3 RC2:<br/> <a href="https://maloo.whamcloud.com/test_sets/1275a262-eb3b-11e1-ba73-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1275a262-eb3b-11e1-ba73-52540035b04c</a></p> <p>Lustre Server Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/32" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/32</a><br/> Lustre Client Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/121" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/121</a><br/> Distribution: CentOS release 6.3<br/> <a href="https://maloo.whamcloud.com/test_sets/d27db876-1285-11e2-a663-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d27db876-1285-11e2-a663-52540035b04c</a></p> <p>Lustre Client: v2_1_4_RC1<br/> Lustre Server: 2.1.3<br/> Distro/Arch: RHEL6.3/x86_64<br/> Network: IB (in-kernel OFED) <br/> <a href="https://maloo.whamcloud.com/test_sets/635a52e8-487b-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/635a52e8-487b-11e2-8cdc-52540035b04c</a></p> <p>I took a good look at maloo today a few things stand out. </p> <p>Master review has not see this issue since last summer but review seem to no run replay_dual since about the same time. </p> <p>The error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_0a failed with 1 </pre> </div></div> <p>Looks like it might be the same same error as it reports</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>stat: cannot read file system information for `/mnt/lustre': Resource temporarily unavailable replay-dual test_0a: @@@@@@ FAIL: test_0a failed with 1 </pre> </div></div> <p>And we know what part of the problem was there was some</p> <p>I can't seem to find a spot where this has failed on master in a while. It for sure plagues 2.1,2.2 and 2.3 but I don't see many signs of issues on present day Master.</p> <p>The more recent:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>gethostbyname("hostname") failed </pre> </div></div> <p>Errors were fixed by: <a href="https://jira.whamcloud.com/browse/LU-2008" title="After hardware reboot (using pm) the node cannot be accessed" class="issue-link" data-issue-key="LU-2008"><del>LU-2008</del></a></p> <p>There is no sign of the lu-482 skip that the test makes possible. </p> <p>The test reports 7 out of 100. Those 7 are "full" tests running interopt with 2.1 </p> <p>Was the source of the corruption of the group descriptors ever nailed down? I am seeing something similar in some failover testing on 2.3.</p> <p>To add to <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=kelsey" class="user-hover" rel="kelsey">kelsey</a>'s question, the environment we are seeing this in <b>is</b> also VM clusters however the block devices in the VMs that are used for Lustre target are <b>not</b> on LVs, but are directly on the virtual disks provided by KVM.</p> <p>This appears to be the reason that replay-dual was first disabled. The recent test runs don't show any failures in this test.</p> <p>This does not appear to be failing anymore.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14414" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14414</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> tests: enable replay-dual in default test runs<br/> Project: private/autotest<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 55dc7a2e14f9722fee46e4a635f892b498ea51f2</p> <p>Pushed a patch under <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a> to disable test_26 so that this can be moved back into review testing.</p> Related LU-7372 LU-6183 LU-1779 Development Rank 1|hzv6iv: Rank (Obsolete) 4591 Severity