https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2008 Lustre <p>This issue was created by maloo for bobijam &lt;bobijam@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/02a8d976-05c6-11e2-b6a7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/02a8d976-05c6-11e2-b6a7-52540035b04c</a>.</p> <p>The sub-test test_0b failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: replay-ost-single 0b</p> <p>11:42:23:== replay-ost-single test 0b: empty replay =========================================================== 11:42:21 (1348425741)<br/> 11:42:34:Failing ost1 on node client-25vm4<br/> 11:42:34:CMD: client-25vm4 lctl dl<br/> 11:42:34:CMD: client-25vm4 lctl dl<br/> 11:42:34:CMD: client-25vm4 lctl dl<br/> 11:42:34:CMD: client-25vm4 lctl dl<br/> 11:42:34:CMD: client-25vm4 lctl dl<br/> 11:42:34:CMD: client-25vm4 lctl dl<br/> 11:42:34:CMD: client-25vm4 lctl dl<br/> 11:42:34:+ pm -h powerman --off client-25vm4<br/> 11:42:34:pm: warning: server version (2.3.5) != client (2.3.12)<br/> 11:42:34:Command completed successfully<br/> 11:42:34:affected facets:<br/> 11:42:34:+ pm -h powerman --on client-25vm4<br/> 11:42:34:pm: warning: server version (2.3.5) != client (2.3.12)<br/> 11:42:46:Command completed successfully<br/> 11:42:46:CMD: hostname <br/> 11:42:46:pdsh@client-25vm1: gethostbyname("hostname") failed<br/> 11:42:46:CMD: hostname <br/> 11:42:46:pdsh@client-25vm1: gethostbyname("hostname") failed<br/> 11:42:57:CMD: hostname <br/> 11:42:57:pdsh@client-25vm1: gethostbyname("hostname") failed<br/> 11:42:57:CMD: hostname </p> LU-2008

After hardware reboot (using pm) the node cannot be accessed

Bug Blocker Resolved Fixed Jian Yu Maloo HB Sun, 23 Sep 2012 20:24:11 +0000 Tue, 15 Oct 2013 13:52:48 +0000 Mon, 8 Apr 2013 15:53:46 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.5.0 0 10 <p>Chris,</p> <p>Does it related to TT-868, or another issue? The test failed over an OSS (client-25vm4) and lost it forever.</p> <p>test parameter are: </p> <p>Test-Parameters: fortestonly envdefinitions=SLOW=yes clientcount=4 osscount=2 mdscount=2 austeroptions=-R failover=true useiscsi=true testlist=replay-ost-single</p> <p>This is blocking the work on <a href="https://jira.whamcloud.com/browse/LU-1966" title="Test failure on test suite replay-ost-single, subtest test_6: Destroys weren&#39;t done in 5 sec" class="issue-link" data-issue-key="LU-1966"><del>LU-1966</del></a> which is also a 2.3 blocker.</p> <p>Dropping priority as this is not preventing any testing from happening</p> <p>hit this issue again during 2.4 testing:</p> <p><a href="https://maloo.whamcloud.com/test_sessions/6e837946-4809-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/6e837946-4809-11e2-8cdc-52540035b04c</a></p> <p>Well I saw <a href="https://jira.whamcloud.com/browse/LU-2599" title="Failure on test suite recovery-random-scale test_fail_client_mds: gethostbyname failed" class="issue-link" data-issue-key="LU-2599"><del>LU-2599</del></a> and that it was duped here. </p> <p>from <a href="https://maloo.whamcloud.com/test_sets/8b557c90-4809-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8b557c90-4809-11e2-8cdc-52540035b04c</a> <br/> recovery-random-scale</p> <p>I took a quick look at the test failure and two things really pop out at me.</p> <p>"Duration: 86400s " My calculator tell me that is 24 hours. </p> <p>"Failure Rate: 78.00% of last 100 executions <span class="error">&#91;all branches&#93;</span> " I am learning to not trust this Metric but as testing is taking days right now this could be part of the issue.</p> <p>From the syslog of the test session:</p> <p>The Start:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Dec 15 02:39:30 fat-intel-3vm5 kernel: Lustre: DEBUG MARKER: /usr/sbin/lctl mark Starting failover on mds1 Dec 15 02:39:35 fat-intel-3vm5 xinetd[1587]: EXIT: shell status=0 pid=7402 duration=5(sec) Dec 15 02:40:33 fat-intel-3vm5 xinetd[1587]: START: shell pid=7430 from=::ffff:10.10.4.86 Dec 15 02:40:33 fat-intel-3vm5 rshd[7431]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD=yes MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "/usr/sbin/lctl mark Starting failover on mds1");echo XXRETCODE:$?' Dec 15 02:40:33 fat-intel-3vm5 kernel: Lustre: DEBUG MARKER: Starting failover on mds1 Dec 15 02:40:33 fat-intel-3vm5 xinetd[1587]: EXIT: shell status=0 pid=7430 duration=0(sec) Dec 15 02:41:20 fat-intel-3vm5 kernel: LustreError: 7107:0:(vvp_io.c:1075:vvp_io_commit_write()) Write page 465052 of inode ffff8800765bd638 failed -28 Dec 15 02:43:50 fat-intel-3vm5 kernel: LustreError: 7459:0:(vvp_io.c:1075:vvp_io_commit_write()) Write page 465147 of inode ffff8800765bd638 failed -28 Dec 15 02:43:51 fat-intel-3vm5 kernel: LustreError: 7471:0:(vvp_io.c:1075:vvp_io_commit_write()) Write page 0 of inode ffff8800765bd638 failed -28 Dec 15 02:46:20 fat-intel-3vm5 kernel: LustreError: 7475:0:(vvp_io.c:1075:vvp_io_commit_write()) Write page 464979 of inode ffff8800765bd638 failed -28 Dec 15 02:48:50 fat-intel-3vm5 kernel: LustreError: 7487:0:(vvp_io.c:1075:vvp_io_commit_write()) Write page 465103 of inode ffff8800765bd638 failed -28 </pre> </div></div> <p>A day or so of write page failed errors... </p> <p>The end: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Dec 16 02:33:30 fat-intel-3vm5 kernel: LustreError: 6617:0:(vvp_io.c:1075:vvp_io_commit_write()) Write page 464781 of inode ffff8800765bd638 failed -28 Dec 16 02:33:30 fat-intel-3vm5 kernel: LustreError: 6617:0:(vvp_io.c:1075:vvp_io_commit_write()) Skipped 5 previous similar messages Dec 16 02:39:32 fat-intel-3vm5 xinetd[1587]: START: shell pid=6666 from=::ffff:10.10.4.86 Dec 16 02:39:32 fat-intel-3vm5 rshd[6667]: autotest@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/home/autotest/.autotest/dynamic_bash/70041011263340' Dec 16 02:39:34 fat-intel-3vm5 kernel: SysRq : Show State Dec 16 02:39:34 fat-intel-3vm5 kernel: task PC stack pid father </pre> </div></div> <p>I am going to dig into this a bit more but something seems really broken here. </p> <p>Seems to be a script var issues:</p> <p>from The test output (just part of it)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>FAIL CLIENT fat-intel-3vm6... + pm -h powerman --off fat-intel-3vm6 pm: warning: server version (2.3.5) != client (2.3.12) Command completed successfully Starting failover on mds1 CMD: fat-intel-3vm3 /usr/sbin/lctl dl Failing on + pm -h powerman --off Usage: pm [action] [targets] -1,--on targets Power on targets -0,--off targets Power off targets -c,--cycle targets Power cycle targets </pre> </div></div> <p>So we fail Client fat-intel-3vm6 just fine.</p> <p>Then we start on the MDS "Starting failover on mds1"</p> <p>We get "Failing on " There is nothing here. </p> <p>pm is confused and sending out it usage info because we don't pass in a host. </p> <p>From lustre/tests/recovery-random-scale.sh calles</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> log "Starting failover on $serverfacet" facet_failover "$serverfacet" || exit 1 </pre> </div></div> <p>In test-framework.sh facet_failover</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> for ((index=0; index&lt;$total; index++)); do facet=$(echo ${affecteds[index]} | tr -s " " | cut -d"," -f 1) local host=$(facet_active_host $facet) echo "Failing ${affecteds[index]} on $host" shutdown_facet $facet done </pre> </div></div> <p>$host is not set and ${affecteds<span class="error">&#91;index&#93;</span>} is empty.</p> <p>Something in the framework is not quite right and I don't think I am quite the right person to dig all that out.</p> <p>I will put a sanity check into shutdown_facet() to keep errors like this from wasting a day of cycles.</p> <p>Well with the lack of return codes and argument it makes is reasonably complicated to properly detect the error and skip timeout. </p> <p>I noticed there were widespread timout issues with the total run. 7 tests goto timeout. </p> <p><a href="http://jira.whamcloud.com/browse/TT-1016" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/TT-1016</a> Seems very related. </p> <p>The root issue may be occurring at a reasonably high rate and it is eating up lots of cycles on the test cluster. </p> <p>When things get off the rails like this we need to have a good way of bailing out. </p> <p>Below is the configure used by one of the runs. Looks out to me but someone else might like to review and work out what is wrong.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#!/bin/bash #Auto Generated By Whamcloud Autotest #Key Exports export mgs_HOST=fat-intel-3vm3 export mds_HOST=fat-intel-3vm3 export MGSDEV=/dev/lvm-MDS/P1 export MDSDEV=/dev/lvm-MDS/P1 export mds1_HOST=fat-intel-3vm3 export MDSDEV1=/dev/lvm-MDS/P1 export MDSCOUNT=1 export MDSSIZE=10485760 export MGSSIZE=10485760 export MDSFSTYPE=ldiskfs export MGSFSTYPE=ldiskfs export mdsfailover_HOST=fat-intel-3vm7 export mds1failover_HOST=fat-intel-3vm7 export MGSNID=fat-intel-3vm3:fat-intel-3vm7 export FAILURE_MODE=HARD export POWER_DOWN=<span class="code-quote">"pm -h powerman --off"</span> export POWER_UP=<span class="code-quote">"pm -h powerman --on"</span> export ost_HOST=fat-intel-3vm4 export ostfailover_HOST=fat-intel-3vm8 export ost1_HOST=fat-intel-3vm4 export OSTDEV1=/dev/lvm-OSS/P1 export ost1failover_HOST=fat-intel-3vm8 export ost2_HOST=fat-intel-3vm4 export OSTDEV2=/dev/lvm-OSS/P2 export ost2failover_HOST=fat-intel-3vm8 export ost3_HOST=fat-intel-3vm4 export OSTDEV3=/dev/lvm-OSS/P3 export ost3failover_HOST=fat-intel-3vm8 export ost4_HOST=fat-intel-3vm4 export OSTDEV4=/dev/lvm-OSS/P4 export ost4failover_HOST=fat-intel-3vm8 export ost5_HOST=fat-intel-3vm4 export OSTDEV5=/dev/lvm-OSS/P5 export ost5failover_HOST=fat-intel-3vm8 export ost6_HOST=fat-intel-3vm4 export OSTDEV6=/dev/lvm-OSS/P6 export ost6failover_HOST=fat-intel-3vm8 export ost7_HOST=fat-intel-3vm4 export OSTDEV7=/dev/lvm-OSS/P7 export ost7failover_HOST=fat-intel-3vm8 # some setup <span class="code-keyword">for</span> conf-sanity test 24a, 24b, 33a export fs2mds_DEV=/dev/lvm-MDS/S1 export fs2ost_DEV=/dev/lvm-OSS/S1 export fs3ost_DEV=/dev/lvm-OSS/S2 export RCLIENTS=<span class="code-quote">"fat-intel-3vm6 fat-intel-3vm5"</span> export OSTCOUNT=7 export NETTYPE=tcp export OSTSIZE=2097152 export OSTFSTYPE=ldiskfs export FSTYPE=ldiskfs export SHARED_DIRECTORY=/home/autotest/.autotest/shared_dir/2012-12-15/011204-70041009461540 export SLOW=yes # Adding contents of /home/autotest/autotest/mecturk-standalone.sh VERBOSE=<span class="code-keyword">true</span> # Entries above here come are created by configurecluster.rb # Entries below here come from mecturk.h FSNAME=lustre TMP=${TMP:-/tmp} DAEMONSIZE=${DAEMONSIZE:-500} MDSOPT=${MDSOPT:-""} MGSOPT=${MGSOPT:-""} # sgpdd-survey requires these to be set. They apprarently have no side affect. SGPDD_YES=<span class="code-keyword">true</span> REFORMAT=<span class="code-keyword">true</span> # some bits <span class="code-keyword">for</span> liblustre tcp connecttions export LNET_ACCEPT_PORT=7988 export ACCEPTOR_PORT=7988 OSTOPT=${OSTOPT:-""} STRIPE_BYTES=${STRIPE_BYTES:-1048576} STRIPES_PER_OBJ=${STRIPES_PER_OBJ:-0} SINGLEMDS=${SINGLEMDS:-<span class="code-quote">"mds1"</span>} TIMEOUT=${TIMEOUT:-20} PTLDEBUG=${PTLDEBUG:-0x33f0404} DEBUG_SIZE=${DEBUG_SIZE:-32} SUBSYSTEM=${SUBSYSTEM:- 0xffb7e3ff} MKFSOPT="" MOUNTOPT="" [ <span class="code-quote">"x$MDSJOURNALSIZE"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MKFSOPT=$MKFSOPT<span class="code-quote">" -J size=$MDSJOURNALSIZE"</span> [ <span class="code-quote">"x$MDSISIZE"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MKFSOPT=$MKFSOPT<span class="code-quote">" -i $MDSISIZE"</span> [ <span class="code-quote">"x$MKFSOPT"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MKFSOPT=<span class="code-quote">"--mkfsoptions=\\\"</span>$MKFSOPT\\\"" [ <span class="code-quote">"x$MDSCAPA"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MKFSOPT=<span class="code-quote">"--param mdt.capa=$MDSCAPA"</span> [ <span class="code-quote">"$MDSFSTYPE"</span> = <span class="code-quote">"ldiskfs"</span> ] &amp;&amp; MDSOPT=$MDSOPT<span class="code-quote">" --mountfsoptions=errors=remount-ro,iopen_nopriv,user_xattr,acl"</span> [ <span class="code-quote">"x$mdsfailover_HOST"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MDSOPT=$MDSOPT<span class="code-quote">" --failnode=`h2$NETTYPE $mdsfailover_HOST`"</span> [ <span class="code-quote">"x$STRIPE_BYTES"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MOUNTOPT=$MOUNTOPT<span class="code-quote">" --param lov.stripesize=$STRIPE_BYTES"</span> [ <span class="code-quote">"x$STRIPES_PER_OBJ"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MOUNTOPT=$MOUNTOPT<span class="code-quote">" --param lov.stripecount=$STRIPES_PER_OBJ"</span> [ <span class="code-quote">"x$L_GETIDENTITY"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MOUNTOPT=$MOUNTOPT<span class="code-quote">" --param mdt.identity_upcall=$L_GETIDENTITY"</span> # Check <span class="code-keyword">for</span> wide stripping [ $OSTCOUNT -gt 160 ] &amp;&amp; MDSOPT=$MDSOPT<span class="code-quote">" --mkfsoptions=-O large_xattr -J size=4096"</span> MDS_MKFS_OPTS=<span class="code-quote">"--mdt --fsname=$FSNAME $MKFSOPT $MDSOPT"</span> [ <span class="code-quote">"$MDSFSTYPE"</span> = <span class="code-quote">"ldiskfs"</span> ] &amp;&amp; MDS_MKFS_OPTS=$MDS_MKFS_OPTS<span class="code-quote">" --param sys.timeout=$TIMEOUT --device-size=$MDSSIZE"</span> [ <span class="code-quote">"$MDSFSTYPE"</span> = <span class="code-quote">"zfs"</span> ] &amp;&amp; MDS_MKFS_OPTS=$MDS_MKFS_OPTS<span class="code-quote">" --vdev-size=$MDSSIZE"</span> <span class="code-keyword">if</span> combined_mgs_mds ; then [ <span class="code-quote">"$MDSCOUNT"</span> = <span class="code-quote">"1"</span> ] &amp;&amp; MDS_MKFS_OPTS=<span class="code-quote">"--mgs $MDS_MKFS_OPTS"</span> <span class="code-keyword">else</span> MDS_MKFS_OPTS=<span class="code-quote">"--mgsnode=$MGSNID $MDS_MKFS_OPTS"</span> [ <span class="code-quote">"$MGSFSTYPE"</span> = <span class="code-quote">"ldiskfs"</span> ] &amp;&amp; MGS_MKFS_OPTS=<span class="code-quote">"--mgs --device-size=$MGSSIZE"</span> [ <span class="code-quote">"$MGSFSTYPE"</span> = <span class="code-quote">"zfs"</span> ] &amp;&amp; MGS_MKFS_OPTS=<span class="code-quote">"--mgs --vdev-size=$MGSSIZE"</span> fi <span class="code-keyword">if</span> [ <span class="code-quote">"$MDSDEV1"</span> != <span class="code-quote">"$MGSDEV"</span> ]; then <span class="code-keyword">if</span> [ <span class="code-quote">"$MGSFSTYPE"</span> == <span class="code-quote">"ldiskfs"</span> ]; then MGS_MOUNT_OPTS=${MGS_MOUNT_OPTS:-<span class="code-quote">"-o loop"</span>} <span class="code-keyword">else</span> MGS_MOUNT_OPTS=${MGS_MOUNT_OPTS:-""} fi <span class="code-keyword">else</span> MGS_MOUNT_OPTS=${MGS_MOUNT_OPTS:-$MDS_MOUNT_OPTS} fi MKFSOPT="" MOUNTOPT="" [ <span class="code-quote">"x$OSTJOURNALSIZE"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MKFSOPT=$MKFSOPT<span class="code-quote">" -J size=$OSTJOURNALSIZE"</span> [ <span class="code-quote">"x$MKFSOPT"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MKFSOPT=<span class="code-quote">"--mkfsoptions=\\\"</span>$MKFSOPT\\\"" [ <span class="code-quote">"x$OSSCAPA"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MKFSOPT=<span class="code-quote">"--param ost.capa=$OSSCAPA"</span> [ <span class="code-quote">"x$ostfailover_HOST"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; OSTOPT=$OSTOPT<span class="code-quote">" --failnode=`h2$NETTYPE $ostfailover_HOST`"</span> OST_MKFS_OPTS=<span class="code-quote">"--ost --fsname=$FSNAME --mgsnode=$MGSNID $MKFSOPT $OSTOPT"</span> [ <span class="code-quote">"$OSTFSTYPE"</span> = <span class="code-quote">"ldiskfs"</span> ] &amp;&amp; OST_MKFS_OPTS=$OST_MKFS_OPTS<span class="code-quote">" --param sys.timeout=$TIMEOUT --device-size=$OSTSIZE"</span> [ <span class="code-quote">"$OSTFSTYPE"</span> = <span class="code-quote">"zfs"</span> ] &amp;&amp; OST_MKFS_OPTS=$OST_MKFS_OPTS<span class="code-quote">" --vdev-size=$OSTSIZE"</span> MDS_MOUNT_OPTS=${MDS_MOUNT_OPTS:-<span class="code-quote">"-o user_xattr,acl"</span>} OST_MOUNT_OPTS=${OST_MOUNT_OPTS:-""} # TT-430 SERVER_FAILOVER_PERIOD=$((60 * 15)) #RUNAS_ID=840000017 #client MOUNT=${MOUNT:-/mnt/${FSNAME}} MOUNT1=${MOUNT1:-$MOUNT} MOUNT2=${MOUNT2:-${MOUNT}2} MOUNTOPT=${MOUNTOPT:-<span class="code-quote">"-o user_xattr,acl,flock"</span>} [ <span class="code-quote">"x$RMTCLIENT"</span> != <span class="code-quote">"x"</span> ] &amp;&amp; MOUNTOPT=$MOUNTOPT<span class="code-quote">",remote_client"</span> DIR=${DIR:-$MOUNT} DIR1=${DIR:-$MOUNT1} DIR2=${DIR2:-$MOUNT2} <span class="code-keyword">if</span> [ $UID -ne 0 ]; then log <span class="code-quote">"running as non-root uid $UID"</span> RUNAS_ID=<span class="code-quote">"$UID"</span> RUNAS_GID=`id -g $USER` RUNAS="" <span class="code-keyword">else</span> RUNAS_ID=${RUNAS_ID:-500} RUNAS_GID=${RUNAS_GID:-$RUNAS_ID} RUNAS=${RUNAS:-<span class="code-quote">"runas -u $RUNAS_ID"</span>} fi PDSH=<span class="code-quote">"pdsh -t 120 -S -Rrsh -w"</span> #PDSH=<span class="code-quote">"pdsh -t 120 -S -Rmrsh -w"</span> export RSYNC_RSH=rsh FAILURE_MODE=${FAILURE_MODE:-SOFT} # or HARD POWER_DOWN=${POWER_DOWN:-<span class="code-quote">"powerman --off"</span>} POWER_UP=${POWER_UP:-<span class="code-quote">"powerman --on"</span>} SLOW=${SLOW:-no} FAIL_ON_ERROR=${FAIL_ON_ERROR:-<span class="code-keyword">true</span>} # error: conf_param: No such device" issue in every test suite logs # sanity-quota test_32 hash_lqs_cur_bits isnt set properly QUOTA_TYPE=${QUOTA_TYPE:-<span class="code-quote">"ug3"</span>} QUOTA_USERS=${QUOTA_USERS:-<span class="code-quote">"quota_usr quota_2usr sanityusr sanityusr1"</span>} LQUOTAOPTS=${LQUOTAOPTS:-<span class="code-quote">"hash_lqs_cur_bits=3"</span>} # SKIP: parallel-scale test_compilebench compilebench not found # SKIP: parallel-scale test_connectathon connectathon dir not found # ------ cbench_DIR=/usr/bin cnt_DIR=/opt/connectathon MPIRUN=$(which mpirun 2&gt;/dev/<span class="code-keyword">null</span>) || <span class="code-keyword">true</span> MPIRUN_OPTIONS=<span class="code-quote">"-mca boot ssh"</span> MPI_USER=${MPI_USER:-mpiuser} SINGLECLIENT=$(hostname) #cbench_DIR=/data/src/benchmarks/compilebench.hg #cnt_DIR=/data/src/benchmarks/cthon04 # For multiple clients testing, we need use the cfg/ncli.sh config file, and # only need specify the <span class="code-quote">"RCLIENTS"</span> variable. The <span class="code-quote">"CLIENTS"</span> and <span class="code-quote">"CLIENTCOUNT"</span> # variables are defined in init_clients_lists(), which is called from cfg/ncli.sh. # So, <span class="code-keyword">if</span> we add the contents of cfg/ncli.sh into autotest_config.sh, we would not # need specify <span class="code-quote">"CLIENTS"</span> and <span class="code-quote">"CLIENTCOUNT"</span>, and the above two issues (#3 and #4) would also be fixed. # Start of contents of cfg/ncli.sh CLIENT1=${CLIENT1:-`hostname`} SINGLECLIENT=$CLIENT1 RCLIENTS=${RCLIENTS:-""} init_clients_lists [ -n <span class="code-quote">"$RCLIENTS"</span> -a <span class="code-quote">"$PDSH"</span> = <span class="code-quote">"no_dsh"</span> ] &amp;&amp; \ error <span class="code-quote">"tests <span class="code-keyword">for</span> remote clients $RCLIENTS needs pdsh != do_dsh "</span> || <span class="code-keyword">true</span> [ -n <span class="code-quote">"$FUNCTIONS"</span> ] &amp;&amp; . $FUNCTIONS || <span class="code-keyword">true</span> # <span class="code-keyword">for</span> recovery scale tests # <span class="code-keyword">default</span> boulder cluster iozone location export PATH=/opt/iozone/bin:$PATH LOADS=${LOADS:-<span class="code-quote">"dd tar dbench iozone"</span>} <span class="code-keyword">for</span> i in $LOADS; <span class="code-keyword">do</span> [ -f $LUSTRE/tests/run_${i}.sh ] || \ error <span class="code-quote">"incorrect load: $i"</span> done CLIENT_LOADS=($LOADS) # End of contents of cfg/ncli.sh </pre> </div></div> <p>Has anyone set this test up manually and run it?</p> <p>Yu Jian will investigate the problems in this area</p> <p>There is a common issue in lustre-initialization-1 reports in the above test sessions. After formatting all of the server targets, mounting them hit the following issues:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>11:37:12:Setup mgs, mdt, osts 11:37:12:CMD: client-25vm3 mkdir -p /mnt/mds1 11:37:12:CMD: client-25vm3 test -b /dev/lvm-MDS/P1 11:37:12:Starting mds1: -o user_xattr,acl /dev/lvm-MDS/P1 /mnt/mds1 11:37:12:CMD: client-25vm3 mkdir -p /mnt/mds1; mount -t lustre -o user_xattr,acl /dev/lvm-MDS/P1 /mnt/mds1 11:37:43: e2label: MMP: device currently active while trying to open /dev/dm-0 11:37:43: MMP error info: last update: Sun Sep 23 11:37:37 2012 11:37:43: node: client-25vm3.lab.whamcloud.com device: dm-0 11:37:43:CMD: client-25vm3 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"0x33f0404\" \" 0xffb7e3ff\" 32 11:37:43:CMD: client-25vm3 e2label /dev/lvm-MDS/P1 2&gt;/dev/null 11:37:43:Started lustre:MDT0000 11:37:43:CMD: client-25vm4 mkdir -p /mnt/ost1 11:37:43:CMD: client-25vm4 test -b /dev/lvm-OSS/P1 11:37:43:Starting ost1: /dev/lvm-OSS/P1 /mnt/ost1 11:37:43:CMD: client-25vm4 mkdir -p /mnt/ost1; mount -t lustre /dev/lvm-OSS/P1 /mnt/ost1 11:38:14: e2label: MMP: device currently active while trying to open /dev/dm-0 11:38:14: MMP error info: last update: Sun Sep 23 11:38:09 2012 11:38:15: node: client-25vm4.lab.whamcloud.com device: dm-0 11:38:15:CMD: client-25vm4 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/1.4-gcc/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"0x33f0404\" \" 0xffb7e3ff\" 32 11:38:15:CMD: client-25vm4 e2label /dev/lvm-OSS/P1 2&gt;/dev/null 11:38:15:Started lustre:OST0000 </pre> </div></div> <p>The labels of server target devices were "lustre:MDT0000", "lustre:OST0000", etc., instead of "lustre-MDT0000", "lustre-OST0000", which caused facet_up() always return false, and then affected_facets() always returned empty under HARD failure mode.</p> <p><a href="https://jira.whamcloud.com/browse/LU-2415" title="recovery-mds-scale test_failover_mds: lustre:MDT0000/recovery_status found no match" class="issue-link" data-issue-key="LU-2415"><del>LU-2415</del></a> also has the same issue.</p> <p>This is a Lustre issue on master branch. Mounting an ldiskfs server target with MMP feature enabled will fail at ldiskfs_label_lustre() which uses e2label:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fat-amd-2 ~]# mkfs.lustre --mgsnode=client-1@tcp:client-3@tcp --fsname=lustre --ost --index=0 --failnode=fat-amd-3@tcp --param=sys.timeout=20 --backfstype=ldiskfs --device-size=16000000 --quiet --reformat /dev/disk/by-id/scsi-1IET_00020001 Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=10.10.4.1@tcp:10.10.4.3@tcp failover.node=10.10.4.134@tcp sys.timeout=20 [root@fat-amd-2 ~]# e2label /dev/disk/by-id/scsi-1IET_00020001 lustre:OST0000 [root@fat-amd-2 ~]# mkdir -p /mnt/ost1; mount -t lustre /dev/disk/by-id/scsi-1IET_00020001 /mnt/ost1 e2label: MMP: device currently active while trying to open /dev/sdf MMP error info: last update: Wed Mar 27 07:29:05 2013 node: fat-amd-2 device: sdf [root@fat-amd-2 ~]# e2label /dev/disk/by-id/scsi-1IET_00020001 lustre:OST0000 </pre> </div></div> <p>The issue was introduced by <a href="http://review.whamcloud.com/3611" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3611</a>.</p> <p>Patch for master branch is in <a href="http://review.whamcloud.com/5867" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5867</a>.</p> <p>Landed for 2.4</p> Duplicate LU-2415 Related Development Rank 1|hzv3mn: Rank (Obsolete) 4110 Severity