https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2415 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/8106fea4-3a9d-11e2-b2e6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8106fea4-3a9d-11e2-b2e6-52540035b04c</a>.</p> <p>The sub-test test_failover_mds failed with the following error:</p> <blockquote> <p>test_failover_mds returned 7</p></blockquote> <p>test log shows</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== recovery-mds-scale test failover_mds: failover MDS == 13:08:23 (1354136903) Started client load: dd on client-28vm5 CMD: client-28vm5 PATH=/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: MOUNT=/mnt/lustre ERRORS_OK= BREAK_ON_ERROR= END_RUN_FILE=/home/autotest/.autotest/shared_dir/2012-11-28/112808-70113261691540/end_run_file LOAD_PID_FILE=/tmp/client-load.pid TESTLOG_PREFIX=/logdir/test_logs/2012-11-28/lustre-master-el6-x86_64-fo__1065__-70113261691540-112807/recovery-mds-scale TESTNAME=test_failover_mds DBENCH_LIB=/usr/share/doc/dbench/loadfiles DBENCH_SRC= run_dd.sh Started client load: tar on client-28vm6 CMD: client-28vm6 PATH=/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: MOUNT=/mnt/lustre ERRORS_OK= BREAK_ON_ERROR= END_RUN_FILE=/home/autotest/.autotest/shared_dir/2012-11-28/112808-70113261691540/end_run_file LOAD_PID_FILE=/tmp/client-load.pid TESTLOG_PREFIX=/logdir/test_logs/2012-11-28/lustre-master-el6-x86_64-fo__1065__-70113261691540-112807/recovery-mds-scale TESTNAME=test_failover_mds DBENCH_LIB=/usr/share/doc/dbench/loadfiles DBENCH_SRC= run_tar.sh client loads pids: CMD: client-28vm5,client-28vm6 cat /tmp/client-load.pid client-28vm6: 4127 client-28vm5: 4080 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=900 CMD: client-28vm5 rc=\$([ -f /proc/sys/lnet/catastrophe ] &amp;&amp; echo \$(&lt; /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc; CMD: client-28vm5 ps auxwww | grep -v grep | grep -q run_dd.sh CMD: client-28vm6 rc=\$([ -f /proc/sys/lnet/catastrophe ] &amp;&amp; echo \$(&lt; /proc/sys/lnet/catastrophe) || echo 0); if [ \$rc -ne 0 ]; then echo \$(hostname): \$rc; fi exit \$rc; CMD: client-28vm6 ps auxwww | grep -v grep | grep -q run_tar.sh Wait mds1 recovery complete before doing next failover... CMD: client-28vm1.lab.whamcloud.com lctl get_param -n at_max affected facets: mds1 CMD: client-28vm3 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh _wait_recovery_complete *.lustre:MDT0000.recovery_status 662 client-28vm3: error: get_param: /proc/{fs,sys}/{lnet,lustre}/*/lustre:MDT0000/recovery_status: Found no match mds1 recovery is not completed! 2012-11-28 13:08:32 Terminating clients loads ... </pre> </div></div> LU-2415

recovery-mds-scale test_failover_mds: lustre:MDT0000/recovery_status found no match

Bug Minor Closed Duplicate Alex Zhuravlev Maloo Fri, 30 Nov 2012 13:48:16 +0000 Fri, 8 Jul 2016 18:23:54 +0000 Fri, 19 Apr 2013 22:31:27 +0000 Lustre 2.4.0 0 4 <p>It looks like the root of the problem is that it is looking for "<tt>lctl get_param *.lustre:MDT0000.recovery_status</tt>" and "<tt>lctl get_param *.lustre:OST0000.recovery_status</tt>" (note ':' instead of '-' in the device name).</p> <p>Somewhere the test-framework.sh is either finding or caching the wrong device name, or the device name was not updated on disk, or it actually has the wrong name in /proc.</p> <p>just tried with llmount.sh:</p> <p>Setup mgs, mdt, osts<br/> Starting mds1: -o loop /tmp/lustre-mdt1 /mnt/mds1<br/> Started lustre-MDT0000</p> <p>...</p> <ol> <li>debugfs -R stats /tmp/lustre-mdt1 |grep 'volume name'<br/> debugfs 1.42.3.wc3 (15-Aug-2012)<br/> Filesystem volume name: lustre-MDT0000</li> </ol> <p>...</p> <ol> <li>ls /proc/fs/lustre/mdt/<br/> lustre-MDT0000 num_refs</li> </ol> <p>going through test-framework.sh now..</p> <p>The issue was fixed by patch <a href="http://review.whamcloud.com/#change,5867" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5867</a> in <a href="https://jira.whamcloud.com/browse/LU-2008" title="After hardware reboot (using pm) the node cannot be accessed" class="issue-link" data-issue-key="LU-2008"><del>LU-2008</del></a>.</p> Duplicate LU-2008 Related LU-6992 Development Rank 1|hzvd4n: Rank (Obsolete) 5730 Severity