https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9991 Lustre <p>sanity-hsm test_31c fails to archive the newly created file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Update not seen after 200s: wanted 'SUCCEED' got 'STARTED' sanity-hsm test_31c: @@@@@@ FAIL: request on 0x200000402:0x38:0x0 is not SUCCEED on mds1 </pre> </div></div> <p>and then we can shutdown the copytool:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>copytools still running on trevis-51vm6 CMD: trevis-51vm6 pgrep -x lhsmtool_posix trevis-51vm6: 6036 copytools still running on trevis-51vm6 CMD: trevis-51vm6 echo 1 &gt;/proc/sys/kernel/sysrq ; echo t &gt;/proc/sysrq-trigger copytools failed to stop in 200s sanity-hsm test_31c: @@@@@@ FAIL: copytools failed to stop </pre> </div></div> <p>From the copytool logs, the archive was progressing and it doesn’t look like it exceeded 200 seconds unless the copytool was hung:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1505294662.538461 lhsmtool_posix[6038]: '[0x200000402:0x38:0x0]' action ARCHIVE reclen 72, cookie=0x59b8f942 1505294662.539849 lhsmtool_posix[6038]: processing file 'd31c.sanity-hsm/f31c.sanity-hsm' 1505294662.555950 lhsmtool_posix[6038]: archiving '/mnt/lustre2/.lustre/fid/0x200000402:0x38:0x0' to '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' 1505294662.556901 lhsmtool_posix[6038]: saving stripe info of '/mnt/lustre2/.lustre/fid/0x200000402:0x38:0x0' in /tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp.lov 1505294662.558302 lhsmtool_posix[6038]: start copy of 34603008 bytes from '/mnt/lustre2/.lustre/fid/0x200000402:0x38:0x0' to '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' 1505294692.836074 lhsmtool_posix[6038]: %90 1505294692.838435 lhsmtool_posix[6038]: bandwith control: 1048576B/s excess=1048576 sleep for 1.000000000s 1505294695.842294 lhsmtool_posix[6038]: copied 34603008 bytes in 33.285317 seconds 1505294695.896192 lhsmtool_posix[6038]: data archiving for '/mnt/lustre2/.lustre/fid/0x200000402:0x38:0x0' to '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' done 1505294695.896379 lhsmtool_posix[6038]: attr file for '/mnt/lustre2/.lustre/fid/0x200000402:0x38:0x0' saved to archive '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' 1505294695.897327 lhsmtool_posix[6038]: fsetxattr of 'trusted.hsm' on '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' rc=0 (Success) 1505294695.897360 lhsmtool_posix[6038]: fsetxattr of 'trusted.version' on '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' rc=0 (Success) 1505294695.897402 lhsmtool_posix[6038]: fsetxattr of 'trusted.link' on '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' rc=0 (Success) 1505294695.897432 lhsmtool_posix[6038]: fsetxattr of 'trusted.lov' on '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' rc=0 (Success) 1505294695.897476 lhsmtool_posix[6038]: fsetxattr of 'trusted.lma' on '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' rc=0 (Success) 1505294695.897515 lhsmtool_posix[6038]: fsetxattr of 'lustre.lov' on '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' rc=-1 (Operation not supported) 1505294695.897531 lhsmtool_posix[6038]: xattr file for '/mnt/lustre2/.lustre/fid/0x200000402:0x38:0x0' saved to archive '/tmp/arc1/shsm/0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0_tmp' 1505294695.898946 lhsmtool_posix[6038]: symlink '/tmp/arc1/shsm/shadow/d31c.sanity-hsm/f31c.sanity-hsm' to '../../0038/0000/0402/0000/0002/0000/0x200000402:0x38:0x0' done exiting: Terminated </pre> </div></div> <p>So far, we’ve only see this ‘copytool failed to stop’ error for one patch test session. Logs for this failure are at<br/> <a href="https://testing.hpdd.intel.com/test_sets/3e2f0620-9872-11e7-b775-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3e2f0620-9872-11e7-b775-5254006e85c2</a></p> <p>We’ve seen the fail to archive a file several times for this test and the failure has at least once been attributed to <a href="https://jira.whamcloud.com/browse/LU-7988" title="HSM: high lock contention for cdt_llog_lock" class="issue-link" data-issue-key="LU-7988"><del>LU-7988</del></a>.</p> LU-9991

sanity-hsm test 31c fails with 'copytools failed to stop'

Bug Minor Closed Duplicate WC Triage James Nunez Thu, 14 Sep 2017 15:20:11 +0000 Fri, 15 Dec 2017 18:32:19 +0000 Fri, 15 Dec 2017 18:31:54 +0000 0 2 <p>James,<br/> In the auto-test failure logs you have reported, we can see that the copy tool's threads are stuck trying to close the Lustre file :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>09:31:59:[12891.943987] lhsmtool_posix S 000000000000f908 0 6036 1 0x00000080 09:31:59:[12891.945640] ffff88006c5b3b30 0000000000000086 ffff88005e4d3f40 ffff88006c5b3fd8 09:31:59:[12891.947482] ffff88006c5b3fd8 ffff88006c5b3fd8 ffff88005e4d3f40 ffffffff81e87180 09:31:59:[12891.949277] ffff88006c5b3b60 0000000100c07565 ffffffff81e87180 000000000000f908 09:31:59:[12891.950973] Call Trace: 09:31:59:[12891.952275] [&lt;ffffffff816a94c9&gt;] schedule+0x29/0x70 09:31:59:[12891.953752] [&lt;ffffffff816a6f14&gt;] schedule_timeout+0x174/0x2c0 09:31:59:[12891.955315] [&lt;ffffffff81098b20&gt;] ? internal_add_timer+0x70/0x70 09:31:59:[12891.956858] [&lt;ffffffffc09adec0&gt;] ptlrpc_set_wait+0x4c0/0x910 [ptlrpc] 09:31:59:[12891.958477] [&lt;ffffffff810c4810&gt;] ? wake_up_state+0x20/0x20 09:31:59:[12891.959988] [&lt;ffffffffc09ae38d&gt;] ptlrpc_queue_wait+0x7d/0x220 [ptlrpc] 09:31:59:[12891.961644] [&lt;ffffffffc0ac75cc&gt;] mdc_close+0x1bc/0x8a0 [mdc] 09:31:59:[12891.963168] [&lt;ffffffffc0b6f426&gt;] lmv_close+0x216/0x540 [lmv] 09:31:59:[12891.964760] [&lt;ffffffffc0bb4b48&gt;] ll_close_inode_openhandle+0x2f8/0xe20 [lustre] 09:31:59:[12891.966402] [&lt;ffffffffc0bb8270&gt;] ll_md_real_close+0xf0/0x1e0 [lustre] 09:31:59:[12891.968043] [&lt;ffffffffc0bb899d&gt;] ll_file_release+0x63d/0xa70 [lustre] 09:31:59:[12891.969592] [&lt;ffffffffc0ba424f&gt;] ll_dir_release+0x2f/0xd0 [lustre] 09:31:59:[12891.971179] [&lt;ffffffff81202fb9&gt;] __fput+0xe9/0x260 09:31:59:[12891.972581] [&lt;ffffffff8120321e&gt;] ____fput+0xe/0x10 09:31:59:[12891.974049] [&lt;ffffffff810ad247&gt;] task_work_run+0xa7/0xf0 09:31:59:[12891.975494] [&lt;ffffffff8102ab62&gt;] do_notify_resume+0x92/0xb0 09:31:59:[12891.977059] [&lt;ffffffff816b527d&gt;] int_signal+0x12/0x17 09:31:59:[12891.978503] lhsmtool_posix S 000000000000f908 0 6038 1 0x00000080 09:31:59:[12891.980173] ffff88005d3b3b50 0000000000000086 ffff880063d1bf40 ffff88005d3b3fd8 09:31:59:[12891.981897] ffff88005d3b3fd8 ffff88005d3b3fd8 ffff880063d1bf40 ffff88007c100000 09:31:59:[12891.983599] ffff88005d3b3b80 0000000100c04b03 ffff88007c100000 000000000000f908 09:31:59:[12891.985316] Call Trace: 09:31:59:[12891.986518] [&lt;ffffffff816a94c9&gt;] schedule+0x29/0x70 09:31:59:[12891.987982] [&lt;ffffffff816a6f14&gt;] schedule_timeout+0x174/0x2c0 09:31:59:[12891.989450] [&lt;ffffffff81098b20&gt;] ? internal_add_timer+0x70/0x70 09:31:59:[12891.991030] [&lt;ffffffffc09adec0&gt;] ptlrpc_set_wait+0x4c0/0x910 [ptlrpc] 09:31:59:[12891.992610] [&lt;ffffffff810c4810&gt;] ? wake_up_state+0x20/0x20 09:31:59:[12891.994084] [&lt;ffffffffc09ae38d&gt;] ptlrpc_queue_wait+0x7d/0x220 [ptlrpc] 09:31:59:[12891.995683] [&lt;ffffffffc0ac75cc&gt;] mdc_close+0x1bc/0x8a0 [mdc] 09:31:59:[12891.997162] [&lt;ffffffffc0b6f426&gt;] lmv_close+0x216/0x540 [lmv] 09:31:59:[12891.998691] [&lt;ffffffffc0bb4b48&gt;] ll_close_inode_openhandle+0x2f8/0xe20 [lustre] 09:31:59:[12892.000295] [&lt;ffffffffc0bb8270&gt;] ll_md_real_close+0xf0/0x1e0 [lustre] 09:31:59:[12892.001882] [&lt;ffffffffc0bb899d&gt;] ll_file_release+0x63d/0xa70 [lustre] 09:31:59:[12892.003395] [&lt;ffffffff81202fb9&gt;] __fput+0xe9/0x260 09:31:59:[12892.004846] [&lt;ffffffff8120321e&gt;] ____fput+0xe/0x10 09:31:59:[12892.006206] [&lt;ffffffff810ad247&gt;] task_work_run+0xa7/0xf0 09:31:59:[12892.007663] [&lt;ffffffff8102ab62&gt;] do_notify_resume+0x92/0xb0 09:31:59:[12892.009093] [&lt;ffffffff816b527d&gt;] int_signal+0x12/0x17 </pre> </div></div> <p>this explains why the Archive operation is still in Started state and not Succeed due to end of archiving could not be sent by copytool and then why they can't be killed/stopped during sub-test exit.</p> <p>And on the MDS/MDT side we can find the following error message after the archive operation has been started :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>09:25:01:[12442.007026] WARNING: Pool 'lustre-mdt1' has encountered an uncorrectable I/O failure and has been suspended. </pre> </div></div> <p>so my guess is that the problem is on the ZFS/media side.</p> <p>Closing as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-9845" title="ost-pools test_22 hangs with ‘WARNING: Pool &#39;lustre-mdt1&#39; has encountered an uncorrectable I/O failure and has been suspended.’" class="issue-link" data-issue-key="LU-9845"><del>LU-9845</del></a>.</p> Duplicate LU-9845 Development Rank 1|hzzk73: Rank (Obsolete) 9223372036854775807 Severity