https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4613 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/6ac389ea-90ce-11e3-91ee-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/6ac389ea-90ce-11e3-91ee-52540035b04c</a>.</p> <p>The sub-test test_12o failed with the following error:</p> <blockquote> <p>request on 0x200000bd1:0xf:0x0 is not SUCCEED on mds1</p></blockquote> <p>Info required for matching: sanity-hsm 12o</p> client and server: lustre-master build # 1876<br/> client is SLES11 SP3 LU-4613

Failure on test suite sanity-hsm test_12o: request on 0x200000bd1:0xf:0x0 is not SUCCEED on mds1

Bug Minor Resolved Fixed Bruno Faccini Maloo HSM Tue, 11 Feb 2014 22:21:52 +0000 Fri, 21 Feb 2014 13:47:51 +0000 Fri, 21 Feb 2014 13:47:51 +0000 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.6.0 Lustre 2.5.1 0 7 <p>+1 for <a href="https://maloo.whamcloud.com/test_sets/e17437d2-934f-11e3-9f1b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e17437d2-934f-11e3-9f1b-52540035b04c</a>, during auto-tests session for patch <a href="http://review.whamcloud.com/9212" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9212</a> (b2_5 version for <a href="https://jira.whamcloud.com/browse/LU-3834" title="hsm_cdt_request_completed() may clear HS_RELEASED on failed restore" class="issue-link" data-issue-key="LU-3834"><del>LU-3834</del></a>).</p> <p>sanity-hsm/test_12o (I introduced for <a href="https://jira.whamcloud.com/browse/LU-3834" title="hsm_cdt_request_completed() may clear HS_RELEASED on failed restore" class="issue-link" data-issue-key="LU-3834"><del>LU-3834</del></a> !!…) reports a failure waiting for the success of the 2nd diff command (ie, when fault-injection of layouts-swap has been disabled) for the same file/FID, and we can see the following traces in the tests log :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ……… CMD: client-12vm3 lctl set_param fail_loc=0x152 fail_loc=0x152 CMD: client-12vm3 /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA mdt.lustre-MDT0000.hsm.policy=+NRA diff: /mnt/lustre/d12o.sanity-hsm/f12o.sanity-hsm: No data available CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA mdt.lustre-MDT0000.hsm.policy=-NRA CMD: client-12vm3 lctl set_param fail_loc=0 fail_loc=0 CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= Waiting 100 secs for update CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= Waiting 90 secs for update CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= Changed after 17s: from 'FAILED SUCCEED' to '' CMD: client-12vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000bd1:0xf:0x0'.*action='RESTORE'/ {print \$13}' | cut -f2 -d= ………. </pre> </div></div> <p>so, seems that since it searches for hsm operation result for the same FID than a previous failed operation that has may not have been purge from the log, it gets puzzled when receiving both old/new results!</p> <p>Here is sanity-hsm/test_12o current code :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_12o() { # test needs a running copytool copytool_setup mkdir -p $DIR/$tdir local f=$DIR/$tdir/$tfile local fid=$(copy_file /etc/hosts $f) $LFS hsm_archive --archive $HSM_ARCHIVE_NUMBER $f wait_request_state $fid ARCHIVE SUCCEED $LFS hsm_release $f || error "release of $f failed" #define OBD_FAIL_MDS_HSM_SWAP_LAYOUTS 0x152 do_facet $SINGLEMDS lctl set_param fail_loc=0x152 # set no retry action mode cdt_set_no_retry diff -q /etc/hosts $f local st=$? # we check we had a restore failure wait_request_state $fid RESTORE FAILED [[ $st -eq 0 ]] &amp;&amp; error "Restore must fail" # remove no retry action mode cdt_clear_no_retry # check file is still released check_hsm_flags $f "0x0000000d" # retry w/o failure injection do_facet $SINGLEMDS lctl set_param fail_loc=0 diff -q /etc/hosts $f st=$? # we check we had a restore done wait_request_state $fid RESTORE SUCCEED [[ $st -eq 0 ]] || error "Restored file differs" copytool_cleanup } run_test 12o "Layout-swap failure during Restore leaves file released" </pre> </div></div> <p>I will push a patch to introduce a "cdt_purge + wait_for_grace_delay" between both the failing (due to fault-injection) and succeeding (normal conditions) RESTORE attempts, to allow old/1st result to disapear from the log. Current grace_delay is set to 10s by sanity-hsm which is too long between both RESTORE requests here.</p> <p>Patch is at <a href="http://review.whamcloud.com/9235" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9235</a>.</p> <p>While testing patch <a href="http://review.whamcloud.com/9288" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9288</a> on Lustre b2_5 branch, the same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/a20caf5e-984d-11e3-8a38-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a20caf5e-984d-11e3-8a38-52540035b04c</a></p> <p>As per Bruno, this regression failure was introduced by the patch for <a href="https://jira.whamcloud.com/browse/LU-3834" title="hsm_cdt_request_completed() may clear HS_RELEASED on failed restore" class="issue-link" data-issue-key="LU-3834"><del>LU-3834</del></a> landed on Lustre b2_5 build #25.</p> <p>So, the failure is also needed to be fixed on Lustre b2_5 branch.</p> <p>Patch for b2_5 branch: <a href="http://review.whamcloud.com/9295" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9295</a></p> <p>hit again:<br/> <a href="https://maloo.whamcloud.com/test_sets/d436c73c-9855-11e3-8828-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d436c73c-9855-11e3-8828-52540035b04c</a></p> <p>Landed for 2.5.1 and 2.6</p> Development Rank 1|hzweu7: Rank (Obsolete) 12623 Severity