Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-5474] Test failure sanity-hsm test_90: requests did not complete https://jira.whamcloud.com/browse/LU-5474 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run:<br/> <a href="https://testing.hpdd.intel.com/test_sets/90f1f8a2-1ff0-11e4-8610-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/90f1f8a2-1ff0-11e4-8610-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/29b71c62-020e-11e4-a47c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/29b71c62-020e-11e4-a47c-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/e775fca2-021b-11e4-9435-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e775fca2-021b-11e4-9435-5254006e85c2</a></p> <p>The sub-test test_90 failed with the following error:</p> <blockquote> <p>requests did not complete</p></blockquote> <p>Info required for matching: sanity-hsm 90</p> LU-5474 Test failure sanity-hsm test_90: requests did not complete Bug Blocker Resolved Fixed James Nunez Maloo Mon, 11 Aug 2014 20:12:50 +0000 Sun, 3 Apr 2016 03:32:31 +0000 Fri, 5 Dec 2014 13:51:03 +0000 Lustre 2.7.0 Lustre 2.5.4 Lustre 2.7.0 Lustre 2.5.4 0 13 <p>Looks like sanity-hsm/test_90 is also a good candidate to also use a local (ie, non-NFS) HSM root/repository, like sanity-hsm/test_40 for <a href="https://jira.whamcloud.com/browse/LU-3939" title="Test failure on test suite sanity-hsm, subtest test_40" class="issue-link" data-issue-key="LU-3939"><del>LU-3939</del></a> !!</p> <p>In fact I had a more indeed look in one of these sanity-hsm/test_90 latest failures, and even if this sub-test is definitely a good candidate for the same patch than <a href="https://jira.whamcloud.com/browse/LU-3939" title="Test failure on test suite sanity-hsm, subtest test_40" class="issue-link" data-issue-key="LU-3939"><del>LU-3939</del></a> (use a local vs NFS copytool repository), seems it also suffer some issues causing copytool archiving tasks to (momentarily?) hung for a few files.</p> <p>One of the latest sanity-hsm/test_90 failure at <a href="https://testing.hpdd.intel.com/sub_tests/381f4220-24ff-11e4-bd77-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/381f4220-24ff-11e4-bd77-5254006e85c2</a> shows that 2 files in the set to be archived never completed on time :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' Changed after 70s: from 'lrh=[type=10680000 len=136 idx=1/1866] fid=[0x200000401:0x1db:0x0] dfid=[0x200000401:0x1db:0x0] compound/cookie=0x53eecd80/0x53eecd97 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=1/1885] fid=[0x200000401:0x1ee:0x0] dfid=[0x200000401:0x1ee:0x0] compound/cookie=0x53eecd80/0x53eecdaa action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=1/1890] fid=[0x200000401:0x1f3:0x0] dfid=[0x200000401:0x1f3:0x0] compound/cookie=0x53eecd80/0x53eecdaf action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=1/1894] fid=[0x200000401:0x1f6:0x0] dfid=[0x200000401:0x1f6:0x0] compound/cookie=0x53eecd80/0x53eecdb2 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]' to 'lrh=[type=10680000 len=136 idx=1/1866] fid=[0x200000401:0x1db:0x0] dfid=[0x200000401:0x1db:0x0] compound/cookie=0x53eecd80/0x53eecd97 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=1/1894] fid=[0x200000401:0x1f6:0x0] dfid=[0x200000401:0x1f6:0x0] compound/cookie=0x53eecd80/0x53eecdb2 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]' Waiting 30 secs for update CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' Waiting 20 secs for update CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' Waiting 10 secs for update CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: onyx-51vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' Update not seen after 100s: wanted '' got 'lrh=[type=10680000 len=136 idx=1/1866] fid=[0x200000401:0x1db:0x0] dfid=[0x200000401:0x1db:0x0] compound/cookie=0x53eecd80/0x53eecd97 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=1/1894] fid=[0x200000401:0x1f6:0x0] dfid=[0x200000401:0x1f6:0x0] compound/cookie=0x53eecd80/0x53eecdb2 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]' sanity-hsm test_90: @@@@@@ FAIL: requests did not complete Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4526:error_noexit() = /usr/lib64/lustre/tests/test-framework.sh:4557:error() = /usr/lib64/lustre/tests/sanity-hsm.sh:678:wait_all_done() = /usr/lib64/lustre/tests/sanity-hsm.sh:2990:test_90() = /usr/lib64/lustre/tests/test-framework.sh:4819:run_one() = /usr/lib64/lustre/tests/test-framework.sh:4854:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:4706:run_test() = /usr/lib64/lustre/tests/sanity-hsm.sh:2998:main() Dumping lctl log to /logdir/test_logs/2014-08-15/lustre-reviews-el6-x86_64--review-zfs--2_4_1__25865__-70345348471240-053131/sanity-hsm.test_90.*.1408159245.log CMD: onyx-51vm3,onyx-51vm4,onyx-51vm5,onyx-51vm6.onyx.hpdd.intel.com /usr/sbin/lctl dk &gt; /logdir/test_logs/2014-08-15/lustre-reviews-el6-x86_64--review-zfs--2_4_1__25865__-70345348471240-053131/sanity-hsm.test_90.debug_log.\$(hostname -s).1408159245.log; dmesg &gt; /logdir/test_logs/2014-08-15/lustre-reviews-el6-x86_64--review-zfs--2_4_1__25865__-70345348471240-053131/sanity-hsm.test_90.dmesg.\$(hostname -s).1408159245.log </pre> </div></div> <p>Looks like both files archiving stopped at the "symlink" step :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bfaccini-mac01:~ bfaccini$ grep 0x200000401:0x1db:0x0 Downloads//show_text.txt lhsmtool_posix[31299]: '[0x200000401:0x1db:0x0]' action ARCHIVE reclen 72, cookie=0x53eecd97 lhsmtool_posix[31299]: archiving '/mnt/lustre/.lustre/fid/0x200000401:0x1db:0x0' to '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' lhsmtool_posix[31299]: saving stripe info of '/mnt/lustre/.lustre/fid/0x200000401:0x1db:0x0' in /shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp.lov lhsmtool_posix[31299]: going to copy data from '/mnt/lustre/.lustre/fid/0x200000401:0x1db:0x0' to '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' lhsmtool_posix[31299]: data archiving for '/mnt/lustre/.lustre/fid/0x200000401:0x1db:0x0' to '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' done lhsmtool_posix[31299]: attr file for '/mnt/lustre/.lustre/fid/0x200000401:0x1db:0x0' saved to archive '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' lhsmtool_posix[31299]: fsetxattr of 'trusted.hsm' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31299]: fsetxattr of 'trusted.version' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31299]: fsetxattr of 'trusted.link' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31299]: fsetxattr of 'trusted.lov' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31299]: fsetxattr of 'trusted.lma' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31299]: fsetxattr of 'lustre.lov' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31299]: xattr file for '/mnt/lustre/.lustre/fid/0x200000401:0x1db:0x0' saved to archive '/shared_test/2014-08-15/053137-70345348471240/arc1/01db/0000/0401/0000/0002/0000/0x200000401:0x1db:0x0_tmp' bfaccini-mac01:~ bfaccini$ bfaccini-mac01:~ bfaccini$ grep 0x200000401:0x1f6:0x0 Downloads//show_text.txt lhsmtool_posix[31326]: '[0x200000401:0x1f6:0x0]' action ARCHIVE reclen 72, cookie=0x53eecdb2 lhsmtool_posix[31326]: archiving '/mnt/lustre/.lustre/fid/0x200000401:0x1f6:0x0' to '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' lhsmtool_posix[31326]: saving stripe info of '/mnt/lustre/.lustre/fid/0x200000401:0x1f6:0x0' in /shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp.lov lhsmtool_posix[31326]: going to copy data from '/mnt/lustre/.lustre/fid/0x200000401:0x1f6:0x0' to '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' lhsmtool_posix[31326]: data archiving for '/mnt/lustre/.lustre/fid/0x200000401:0x1f6:0x0' to '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' done lhsmtool_posix[31326]: attr file for '/mnt/lustre/.lustre/fid/0x200000401:0x1f6:0x0' saved to archive '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' lhsmtool_posix[31326]: fsetxattr of 'trusted.hsm' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31326]: fsetxattr of 'trusted.version' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31326]: fsetxattr of 'trusted.link' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31326]: fsetxattr of 'trusted.lov' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31326]: fsetxattr of 'trusted.lma' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31326]: fsetxattr of 'lustre.lov' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31326]: xattr file for '/mnt/lustre/.lustre/fid/0x200000401:0x1f6:0x0' saved to archive '/shared_test/2014-08-15/053137-70345348471240/arc1/01f6/0000/0401/0000/0002/0000/0x200000401:0x1f6:0x0_tmp' bfaccini-mac01:~ bfaccini$ </pre> </div></div> <p>vs for a completed file/archiving :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bfaccini-mac01:~ bfaccini$ grep 0x200000401:0x1f0:0x0 Downloads//show_text.txt lhsmtool_posix[31320]: '[0x200000401:0x1f0:0x0]' action ARCHIVE reclen 72, cookie=0x53eecdac lhsmtool_posix[31320]: archiving '/mnt/lustre/.lustre/fid/0x200000401:0x1f0:0x0' to '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' lhsmtool_posix[31320]: saving stripe info of '/mnt/lustre/.lustre/fid/0x200000401:0x1f0:0x0' in /shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp.lov lhsmtool_posix[31320]: going to copy data from '/mnt/lustre/.lustre/fid/0x200000401:0x1f0:0x0' to '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' lhsmtool_posix[31320]: data archiving for '/mnt/lustre/.lustre/fid/0x200000401:0x1f0:0x0' to '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' done lhsmtool_posix[31320]: attr file for '/mnt/lustre/.lustre/fid/0x200000401:0x1f0:0x0' saved to archive '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' lhsmtool_posix[31320]: fsetxattr of 'trusted.hsm' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31320]: fsetxattr of 'trusted.version' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31320]: fsetxattr of 'trusted.link' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31320]: fsetxattr of 'trusted.lov' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31320]: fsetxattr of 'trusted.lma' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31320]: fsetxattr of 'lustre.lov' on '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' rc=-1 (Operation not supported) lhsmtool_posix[31320]: xattr file for '/mnt/lustre/.lustre/fid/0x200000401:0x1f0:0x0' saved to archive '/shared_test/2014-08-15/053137-70345348471240/arc1/01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0_tmp' lhsmtool_posix[31320]: symlink '/shared_test/2014-08-15/053137-70345348471240/arc1/shadow/d90.sanity-hsm/f90.sanity-hsm.45' to '../../01f0/0000/0401/0000/0002/0000/0x200000401:0x1f0:0x0' done lhsmtool_posix[31320]: Action completed, notifying coordinator cookie=0x53eecdac, FID=[0x200000401:0x1f0:0x0], hp_flags=0 err=0 lhsmtool_posix[31320]: llapi_hsm_action_end() on '/mnt/lustre/.lustre/fid/0x200000401:0x1f0:0x0' ok (rc=0) bfaccini-mac01:~ bfaccini$ </pre> </div></div> <p>even if this can also be a NFS issue that could be fixed with a similar patch than <a href="https://jira.whamcloud.com/browse/LU-3939" title="Test failure on test suite sanity-hsm, subtest test_40" class="issue-link" data-issue-key="LU-3939"><del>LU-3939</del></a>, I think that more debug needs to occur before to conclude like this.</p> <p>Any progress on this ticket? I hit this failure some times recently:<br/> <a href="https://testing.hpdd.intel.com/test_sets/160c68b8-406e-11e4-873f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/160c68b8-406e-11e4-873f-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/9e0d6aa4-406f-11e4-8931-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9e0d6aa4-406f-11e4-8931-5254006e85c2</a></p> <p>James</p> <p>Could you please look into this ticket?</p> <p>Thanks</p> <p>Peter</p> <p>TEI-2702 is disabling this test.<br/> We will need to create a new TEI ticket to re-enable this test once it has been fixed.</p> <p>Looking at the results from <a href="http://review.whamcloud.com/#/c/12021/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12021/</a> I see that none of the sub tests we run in sanity-hsm require a shared archive. OTOH as the same results show there are some places where we use make_small() but should probably use make_large_for_cancel(). Or the file size used in make small should be increased beyond 2MB. (Or deterministic synchronization should be used.)</p> <p>The first and most important action is to submit a patch to disable this test via ALWAYS_EXCEPT, and get this landed so that it can be re-enabled at the TEI level again so that it can be re-enabled in the patch that is fixing the problem. </p> <p>Patch to disable sanity-hsm test 90: <a href="http://review.whamcloud.com/12136" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12136</a> </p> <p>Ok, once Change, 12136 lands, I will get TEI-2705 completed so that the test will be enabled at the TEI level.</p> <p>It looks like the sanity-hsm test_90 exception was removed in the autotest framework, so it is possible to re-enable this test along with a patch to fix it.</p> <p>Test patch at <a href="http://review.whamcloud.com/#/c/12069/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12069/</a> . </p> <p>This patch is to collect data on sanity-hsm test 90 failures. Looking over previous logs does not point to a problem in the test nor in the Lustre/HSM code. It just looks like archiving 51 files sometimes takes longer than 100 seconds. The patch tests if allowing for more time allows the files to be archived.</p> <p>I recall seeing a patch to move the HSM archive into $TMP instead of the shared directory. That might speed up the testing, as long as the archive doesn't need to be too large. </p> <p>Andreas, yes this is tracked in <a href="https://jira.whamcloud.com/browse/LU-3939" title="Test failure on test suite sanity-hsm, subtest test_40" class="issue-link" data-issue-key="LU-3939"><del>LU-3939</del></a>. And I remember that I already have made a similar+related comment about test_90 there ...</p> <p>Another failure instance on b2_5:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d03f6910-681e-11e4-acbe-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d03f6910-681e-11e4-acbe-5254006e85c2</a></p> <p>Bruno, can you please make a patch to sanity-hsm test_90() to use $TMP for the HSM Archive, as was done with test_40()? Is there any reason not to make all HSM tests use $TMP for the archive by default, and only make some tests use the NFS shared directory if needed (very large archive, or HSM failover testing)? That would speed up all the HSM tests I think, unless it filled up $TMP and caused test failures.</p> <p>Andreas, <br/> I modified <a href="http://review.whamcloud.com/#/c/12069/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12069/</a> to use $TMP for the archive. </p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12069" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12069</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5474" title="Test failure sanity-hsm test_90: requests did not complete" class="issue-link" data-issue-key="LU-5474"><del>LU-5474</del></a> tests: sanity-hsm test_90 use local HSM_ARCHIVE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 9<br/> Commit: 7a0b4705f130f318acc0aaf6297cd4e0068060e5</p> <p>I've updated sanity-hsm test 90 to use a local disk for archive as was implemented in test 40. For the last two runs with this change, test 90 has failed to archive any files from the file list; <a href="https://testing.hpdd.intel.com/test_sessions/5ac85a16-6ee0-11e4-81e7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/5ac85a16-6ee0-11e4-81e7-5254006e85c2</a> and <a href="https://testing.hpdd.intel.com/test_sets/616b778e-6b77-11e4-b1b4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/616b778e-6b77-11e4-b1b4-5254006e85c2</a> .</p> <p>For test 90, it seems that the archive number specified to the copytool is not the same number as the archive requests. The archive number specified in the call to the copytool is #2 and the archive requests are looking for archive #1. From the client test log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: shadow-3vm9 lhsmtool_posix --daemon --hsm-root /tmp/d90.sanity-hsm --archive 2 --bandwidth 1 /mnt/lustre &lt; /dev/null &gt; /logdir/test_logs/2014-11-17/lustre-reviews-el6-x86_64--review-dne-part-2--2_9_1__28407__-70078147694660-145306/sanity-hsm.test_90.copytool2_log.shadow-3vm9.log 2&gt;&amp;1 ... Changed after 8s: from 'lrh=[type=10680000 len=136 idx=1/1846] fid=[0x400000401:0x1c4:0x0] dfid=[0x400000401:0x1c4:0x0] compound/cookie=0x546ab2b6/0x546ab2b6 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=1/1847] fid=[0x400000401:0x1c5:0x0] dfid=[0x400000401:0x1c5:0x0] compound/cookie=0x546ab2b6/0x546ab2b7 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] </pre> </div></div> <p>On the MDS, it complains that it can't find the agent, which is true, archive #2 was started by the copytool:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 14430:0:(mdt_hsm_cdt_agent.c:338:mdt_hsm_agent_send()) lustre-MDT0000: Cannot find agent for archive 1: rc = -11 </pre> </div></div> <p>Yet, test 40 uses essentially the same calls to start the copytool and the archive numbers for this test are consistent. Test 90 archives from a file list and test 40 does not.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12069/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12069/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5474" title="Test failure sanity-hsm test_90: requests did not complete" class="issue-link" data-issue-key="LU-5474"><del>LU-5474</del></a> tests: sanity-hsm test_90 use local HSM_ARCHIVE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8641822bdb248a034bf4065c7e2e60cdb7a47041</p> <p>Landed for 2.7</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12963" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12963</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5474" title="Test failure sanity-hsm test_90: requests did not complete" class="issue-link" data-issue-key="LU-5474"><del>LU-5474</del></a> tests: sanity-hsm test_90 use local HSM_ARCHIVE<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: b8360cf33b18594c765e2fd55e98a591fdc50fdb</p> <p>The failure occurred frequently recently on Lustre b2_5 branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b9a2d466-8327-11e4-aa2f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b9a2d466-8327-11e4-aa2f-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/0e7e5682-8318-11e4-9195-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0e7e5682-8318-11e4-9195-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/3e1b4c7e-830e-11e4-a45f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3e1b4c7e-830e-11e4-a45f-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/28e422d4-8414-11e4-84b4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/28e422d4-8414-11e4-84b4-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12963/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12963/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5474" title="Test failure sanity-hsm test_90: requests did not complete" class="issue-link" data-issue-key="LU-5474"><del>LU-5474</del></a> tests: sanity-hsm test_90 use local HSM_ARCHIVE<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: 54a78152b70b54ec574774eef97dcf4c27b06d5b</p> Related LU-3939 Development Rank 1|hzwtef: Rank (Obsolete) 15251 Severity