https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1275 Lustre <p>My acc-sm set-ups has been used in testing 1.8.5, 1.8.6, and 1.8.7 successfully.<br/> This is the first time I ran acc-sm against 2.1.1.<br/> The SANITY and SANITYN passed, but all tests in REPLAY_SINGLE failed since<br/> "@@@@@@ FAIL: Restart of mds failed".</p> <p>== test 0a: empty replay == 12:05:12<br/> Filesystem 1K-blocks Used Available Use% Mounted on<br/> service360@o2ib:/lustre<br/> 3937056 205112 3531816 6% /mnt/nbp0-1<br/> Failing mds on node service360<br/> Stopping /mnt/mds (opts<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> affected facets: mds<br/> df pid is 13509<br/> Failover mds to service360<br/> 12:05:26 (1333134326) waiting for service360 network 900 secs ...<br/> 12:05:26 (1333134326) network interface is UP<br/> Starting mds: -o errors=panic,acl /dev/sdb1 /mnt/mds<br/> service360: mount.lustre: mount /dev/sdb1 at /mnt/mds failed: Invalid argument<br/> service360: This may have multiple causes.<br/> service360: Are the mount options correct?<br/> service360: Check the syslog for more info.<br/> mount -t lustre /dev/sdb1 /mnt/mds<br/> Start of /dev/sdb1 on mds failed 22<br/> replay-single test_0a: @@@@@@ FAIL: Restart of mds failed!</p> <p>The /var/log/message of the MGS/MDS node showed:<br/> ...<br/> Mar 30 12:05:10 service360 kernel: Lustre: MGC10.151.26.38@o2ib: Reactivating import<br/> Mar 30 12:05:10 service360 kernel: LustreError: 11254:0:(llog_lvfs.c:473:llog_lvfs_next_block()) Invalid llog tail at log id 17/2375643311 offset 14432<br/> Mar 30 12:05:10 service360 kernel: LustreError: 11254:0:(mgs_handler.c:783:mgs_handle()) MGS handle cmd=502 rc=-22<br/> ...<br/> The replay-single.test_0a.debug_log.service360.log.<span class="error">&#91;12&#93;</span> are attached.</p> Server runs centos 6.2, ofed-1.5.4.1, Lustre 2.1.1.<br/> Client runs sles11sp1, ofed-1.5.4.1, Lustre 1.8.6.<br/> MGS/MDS uses the same device. Two OSS&#39;es. Two clients. LU-1275

Lustre 2.1.1 REPLAY_SINGLE test_0a FAIL: Restart of mds failed

Bug Minor Resolved Won't Fix Minh Diep Jay Lan Fri, 30 Mar 2012 17:55:54 +0000 Wed, 5 Mar 2014 19:15:45 +0000 Wed, 5 Mar 2014 19:15:45 +0000 Lustre 2.1.1 Lustre 1.8.6 0 4 <p>Minh</p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>Peter</p> <p>Could you please help on this?<br/> The same test environment worked fine in 1.8.5 and 1.8.6.<br/> One single test failure in 1.8.7 (see <a href="https://jira.whamcloud.com/browse/LU-1246" title="SANITY_QUOTA test_32 failed in cleanup_and_setup_lustre with LOAD_MODULES_REMOTE=true" class="issue-link" data-issue-key="LU-1246"><del>LU-1246</del></a>)<br/> Much more failure between 2.1.1 server and 1.8.7 client, including this one.<br/> Totally not working between 2.1.1 server and 2.1.1 client.</p> <p>I am going to spend time to convert to auster for 2.1.1 server + 2.1.1 client,<br/> but I really need help in evaluating my environment of 2.1.1 server + 1.8.7 client.</p> <p>ok, looking into this</p> <p>can you show me the config file? or local.sh if you modified it</p> <p>The command used in testing was:</p> <ol> <li>ACC_SM_ONLY="REPLAY_SINGLE" NAME=ncli_nas.v3 RCLIENTS="service332" sh acceptance-small.sh</li> </ol> <p>The ncli_nas.v3 will be attached.</p> <p>I accidentally also attached nas.v3.sh. It was a wrapper. The end result was to<br/> run the command I wrote in the previous comment. The configuration file is ncli_nas.v3.</p> <p>thanks. Did you run this on a client that was running 1.8.6?</p> <p>Yes. It was started from service331, a client. All nodes (mds, 2 oss'es and 2 clients) have the same set of configuration.</p> <p>I don't have a system to try it out now. could you manually run "mount -t lustre -o errors=panic,acl /dev/sdb1 /mnt/mds" on the mds to see if it works</p> <p>I know for the fact that "mount -t lustre -o errors=panic,acl /dev/sdb1 /mnt/mds" works because the command has been executed so many times.</p> <p>However, that brought some thought to me. In fact I ran the acceptance-small.sh in a for-loop:</p> <p>for i in SANITY SANITYN REPLAY_SINGLE CONF_SANITY RECOVERY_SMALL REPLAY_OST_SINGLE REPLAY_DUAL INSANITY SANITY_QUOTA LNET_SELFTEST MMP; do<br/> mkdir $TMP/$i<br/> umount /mnt/nbp0-1 /mnt/nbp0-2 1&gt; /dev/null 2&gt;&amp;1<br/> echo run $i &gt;$TMP/${i}/${i}.output 2&gt;&amp;1<br/> case $i in<br/> SANITY|SANITYN|REPLAY_SINGLE|CONF_SANITY|RECOVERY_SMALL|REPLAY_OST_SINGLE|REPLAY_DUAL|INSANITY|LNET_SELFTEST|MMP)<br/> ACC_SM_ONLY="$i" NAME=ncli_nas.v3 RCLIENTS="service332" sh acceptance-small.sh &gt;&gt;$TMP/${i}/${i}.output 2&gt;&amp;1;;<br/> SANITY_QUOTA)<br/> ACC_SM_ONLY="$i" RCLIENTS="service332" MDSSIZE=4000000 OSTSIZE=4000000 NAME=ncli_nas.v3 sh acceptance-small.sh &gt;&gt;$TMP/${i}/${i}.output 2&gt;&amp;1;;<br/> *)<br/> echo "Test $i not supported.";;<br/> esac<br/> done</p> <p>So, by the time the REPLAY_SINGLE is executed, both SANITY and SANITYN has completed. That means it was not the same as starting from ground zero.</p> <p>So, I rebooted all the machines. Ran "mount -t lustre" to make sure it worked.<br/> Umounted it. Then just ran the acceptance-small.sh with REPLAY_SINGLE without executing SANITY and SANITYN first. Well, it succeeded!</p> <p>Now, this wrapper worked when the lustre server is 1.8.6 (or 1.8.7). Any suggestion to make it work when server runs 2.1.1?</p> <p>Since the REPLAY_SINGLE can be executed successfully on a clean environment, you can close this ticket then. I will figure out a way to work around my problem when testing with 2.x servers. Suggestion is welcome <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I need to reproduce this in the lab and investigate the cause. In the mean time, please try this. Add MDSDEV1=/dev/sdb1 in the config file to see if it makes any different. If you don't care to reformat the FS before every test, you could put export REFORMAT=true in the config file.</p> <p>I also suggest you to explore auster script which has an option to send the result back in our maloo result db.</p> <p>I have this line in my configuration file:<br/> export REFORMAT="--reformat"</p> <p>Would it have the same effect as "export REFORMAT=true"?</p> <p>yes</p> <p>Attached two files, cut from /var/log/messages of the mds server between the MARKER of beginning and end of test 0a.</p> <p>The *.FAIL was the run that failed. and The *.PASS was the run that passed.</p> <p>On a second thought I do not feel comfortable to declare this is a test issue (ie, is a problem of test environment setup.) It could also resulted from mds behaving differently in different situations and represents a real problem.</p> <p>We do not know enough to say either way.</p> <p>Jay &#8211; is this still an issue of concern to you?<br/> Is there any further action you'd like us to take?<br/> I'd like to mark this as resolved &#8211; am I OK to go ahead and do that?<br/> Thanks,<br/> ~ jfc.</p> <p>Yes, please. No longer a problem. Thanks!</p> <p>Thank you <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Not clear if this was a test issue &#8211; but time has moved on and it is no longer a problem.</p> Development Rank 1|hzvf4n: Rank (Obsolete) 6096 Severity