https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1362 Lustre <p>Reply-dual test 16 failed:</p> <p>== replay-dual test 16: fail MDS during recovery (3571) == 17:38:52 (1335573532)<br/> Filesystem 1K-blocks Used Available Use% Mounted on<br/> service360@o2ib:/lustre<br/> 3937056 205112 3531816 6% /mnt/nbp0-1<br/> total: 25 creates in 0.04 seconds: 678.21 creates/second<br/> total: 1 creates in 0.00 seconds: 389.26 creates/second<br/> Failing mds1 on node service360<br/> Stopping /mnt/mds1 (opts<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> affected facets: mds1<br/> Failover mds1 to service360<br/> 17:39:07 (1335573547) waiting for service360 network 900 secs ...<br/> 17:39:07 (1335573547) network interface is UP<br/> Starting mds1: -o errors=panic,acl /dev/sdb1 /mnt/mds1<br/> service360: mount.lustre: mount /dev/sdb1 at /mnt/mds1 failed: Invalid argument<br/> service360: This may have multiple causes.<br/> service360: Are the mount options correct?<br/> service360: Check the syslog for more info.<br/> mount -t lustre /dev/sdb1 /mnt/mds1<br/> Start of /dev/sdb1 on mds1 failed 22<br/> replay-dual test_16: @@@@@@ FAIL: Restart of mds1 failed!<br/> Dumping lctl log to /var/acc-sm/test_logs//1335573120/replay-dual.test_16.*.1335573548.log<br/> tar: Removing leading `/' from member names<br/> /var/acc-sm/test_logs//1335573120/replay-dual-1335573548.tar.bz2<br/> FAIL 16 (45s)</p> <p>The "Invalid argument" was about extents, which we do not turn on on MDS.</p> <p>The replay-dual.test_16.dmesg.service360.1335573548.log seemed to suggest the problem<br/> was a corrupted file:<br/> LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts:<br/> LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs.<br/> LDISKFS-fs (sdb1): ldiskfs_check_descriptors: Checksum for group 0 failed (27004!=29265)<br/> LDISKFS-fs (sdb1): group descriptors corrupted!</p> <p><a href="https://jira.whamcloud.com/browse/LU-699" title="replay-dual test_1 fails to remount mdt" class="issue-link" data-issue-key="LU-699"><del>LU-699</del></a> seemed to have encountered a data corruption problem in reply-dual test_1. I applied the patch and rebuilt a lustre server package, but the test still failed.</p> <p>REPLAY_DUAL-16.tgz is attached.</p> <p>The failure is 100% reproducible. </p> <p>Could the data corruption problem caused by trying to fail-over mds to the same node?<br/> In other words, is it a test-case problem or a real problem?</p> Server: 2.1.1 in centos 6.2, kernel 2.6.32-220.4.1.el6, x86_64, lustre server 2.1.1-0.2nasS<br/> Client: 2.1.1 in centos 6.2, unpatched kernel 2.6.32-220.4.1.el6, x86_64. lustre client 2.1.1-0.2nasC<br/> <br/> 1 mds/mgs (service360)<br/> 2 osses (service361,service362)<br/> 2 clients (service333, service334)<br/> <br/> The lustre git repo can be found at <br/> <a href="https://github.com/jlan/lustre-nas/tree/nas-2.1.1">https://github.com/jlan/lustre-nas/tree/nas-2.1.1</a><br/> LU-1362

replay-dual test_16 fails to remount mdt

Bug Minor Resolved Cannot Reproduce Lai Siyao Jay Lan Wed, 2 May 2012 14:47:05 +0000 Fri, 5 Oct 2012 20:31:04 +0000 Fri, 5 Oct 2012 20:31:04 +0000 Lustre 2.1.1 0 3 <p>Lai</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Attached two replay-dual-*.tar.bz2, one for test_14b and the other test_16.</p> <p>Attached two replay-dual-*.tar.bz2, one for test_14b and the other test_16.</p> <p>Sorry I ended up attaching replay-dual-14b.tar.bz2 and replay-dual-16.tar.bz2 multiple times. Please clean the extra copied up.</p> <p>This site formats mds without extents option, but oss with extents option. To get "Invalid argument" errors out of the testing, I added "noextents" to the MDS_MOUNT_OPTIONS and reran the tests.</p> <p>BTW, I always failed on tests 14b, 16, 20, and 21a. I used to think they were caused by the same problem. But it appeared that test 14b and tst 20 have the same failure signature, while test 16 and test 21a share the same. So, I attached test_logs of test 14b and test 16.</p> <p>This looks to be the same issue of <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a>, Jay, are you using VM in your testing env?</p> <p>No, the test system was not running VM.</p> <p>Will it fail if MDS failover node is different node?</p> <p>I do not have an extra machine to be a MDS failover node.</p> <p>Jay, replay_barrier() calls mcreate after syncing target, as looks suspicious, I've a patch <a href="http://review.whamcloud.com/#change,2931" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2931</a>, could you help verify it?</p> <p>Hi Siyao, unfortunately I had to report that the patch did not help.</p> <p>Jay,</p> <p>You didn't happen to be using an iSCSI device as your target did you? Was it a Linux iSCSI target or some vendor?</p> <p>The target was ATA HDS725050KLA360.</p> <p>I set up my test machines and rerun the test.</p> <p>The test passed with 2.1.3 server with both sles11sp1 2.1.3 client and centos6.3 2.1.3 client.</p> <p>The failure I reported was on 2.1.1 centos client.</p> <p>We can close the case.</p> <p>ok thanks Jay!</p> Development Rank 1|hzvh1b: Rank (Obsolete) 6409 Severity