https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12632 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/0c4bfdc4-b860-11e9-a1bd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0c4bfdc4-b860-11e9-a1bd-52540065bddc</a></p> <p>test_90 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-34vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' CMD: trevis-34vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' Update not seen after 100s: wanted '' got 'lrh=[type=10680000 len=136 idx=1/748] fid=[0x200001b71:0x229:0x0] dfid=[0x200001b71:0x229:0x0] compound/cookie=0x0/0x5d494bd8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=1/749] fid=[0x200001b71:0x22a:0x0] dfid=[0x200001b71:0x22a:0x0] compound/cookie=0x0/0x5d494bd9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=1/750] fid=[0x200001b71:0x22b:0x0] dfid=[0x200001b71:0x22b:0x0] compound/cookie=0x0/0x5d494bda action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]' sanity-hsm test_90: @@@@@@ FAIL: requests did not complete </pre> </div></div> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-hsm test_90 - requests did not complete</p> LU-12632

sanity-hsm test_90: FAIL: requests did not complete

Bug Major Resolved Fixed Hongchao Zhang Maloo Tue, 6 Aug 2019 16:52:24 +0000 Thu, 19 Nov 2020 14:34:21 +0000 Thu, 19 Nov 2020 14:34:21 +0000 Lustre 2.13.0 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.12.4 Lustre 2.12.5 Lustre 2.14.0 0 12 <p>+1 on master branch: <a href="https://testing.whamcloud.com/test_sets/39afca24-c5ef-11e9-98c8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/39afca24-c5ef-11e9-98c8-52540065bddc</a></p> <p>+1 on master branch: <a href="https://testing.whamcloud.com/test_sets/b8aec0e0-c843-11e9-a25b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b8aec0e0-c843-11e9-a25b-52540065bddc</a></p> <p>Hongchao</p> <p>Can you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>As per the logs, the HSM_RESTORE operation was slow and didn't complete during 100 seconds,<br/> but there is no obvious issues in the logs.</p> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36101" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36101</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12632" title="sanity-hsm test_90: FAIL: requests did not complete " class="issue-link" data-issue-key="LU-12632"><del>LU-12632</del></a> test: debug patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: faeb584eeb8a2d8c51dbe834f5d8c79ee3781783</p> <p>there are two kinds of failures "requests did not complete".<br/> 1, On LDiskFS<br/> the related HSM archive operations are not started, and it could be caused by the absence of "libtool"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-34vm7 libtool --mode=e pkill -x lhsmtool_posix onyx-34vm7: sh: libtool: command not found CMD: onyx-34vm7 rm -rf /tmp/arc1/sanity-hsm.test_90/ </pre> </div></div> <p>it cause the previous copy tool can't be killed and affect the following copy tool.</p> <p>2, On ZFS<br/> some have the similar "libtool" issue like LDiskFS.<br/> others were caused by the slow HSM Restore operations, it started to show from Jan 10th, 2019<br/> <a href="https://testing.whamcloud.com/sub_tests/f4a00a5a-14fc-11e9-b7d4-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/f4a00a5a-14fc-11e9-b7d4-52540065bddc</a> (zfs 0.7.9, only 1 times)<br/> <a href="https://testing.whamcloud.com/sub_tests/b4fe93f2-528d-11e9-a256-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/b4fe93f2-528d-11e9-a256-52540065bddc</a> (zfs 0.7.12, only 1 times)<br/> <a href="https://testing.whamcloud.com/sub_tests/e6a8aa78-70fb-11e9-a6f9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/e6a8aa78-70fb-11e9-a6f9-52540065bddc</a> (zfs 0.7.13)<br/> <a href="https://testing.whamcloud.com/sub_tests/bfa06a40-c91d-11e9-90ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/bfa06a40-c91d-11e9-90ad-52540065bddc</a> (zfs 0.8.1)</p> <p>How? Its in the spec and deb files for BuildRequires and Requires for the test. How are the images being constructed in the Maloo environment?</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36152" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36152</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12632" title="sanity-hsm test_90: FAIL: requests did not complete " class="issue-link" data-issue-key="LU-12632"><del>LU-12632</del></a> tests: stop running sanity-hsm 90 for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5df30a16128b23977ef92de3ca06ec59a5f531de</p> <p>Example on master branch: <a href="https://testing.whamcloud.com/test_sets/d52265ac-d409-11e9-9fc9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d52265ac-d409-11e9-9fc9-52540065bddc</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/a70981de-0fb2-11ea-bbc3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a70981de-0fb2-11ea-bbc3-52540065bddc</a></p> <p>+1 on master branch: <a href="https://testing.whamcloud.com/test_sets/016d71b8-3c30-11ea-b1e8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/016d71b8-3c30-11ea-b1e8-52540065bddc</a></p> <p>Still failed on master branch: <a href="https://testing.whamcloud.com/test_sets/13b3b0f2-406e-11ea-ac52-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/13b3b0f2-406e-11ea-ac52-52540065bddc</a></p> <p>more on master: </p> <p><a href="https://testing.whamcloud.com/test_sets/7d2382e8-70e4-4b1f-b455-0e9c9b1e1d1b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7d2382e8-70e4-4b1f-b455-0e9c9b1e1d1b</a></p> <p><a href="https://testing.whamcloud.com/test_sets/404f16d2-3b8a-4bd2-8339-5142f548b657" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/404f16d2-3b8a-4bd2-8339-5142f548b657</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/5ce24eb6-6fb9-404c-8bb0-0c87e9f2cede" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5ce24eb6-6fb9-404c-8bb0-0c87e9f2cede</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/070d81f7-e5a2-4fa3-8b8c-67a7820e29be" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/070d81f7-e5a2-4fa3-8b8c-67a7820e29be</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/8f0417bf-f011-4a15-a624-3c06ff83bc1d" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8f0417bf-f011-4a15-a624-3c06ff83bc1d</a></p> <p>Still a problem?</p> <p>John L. Hammond (jhammond@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40387" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40387</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12632" title="sanity-hsm test_90: FAIL: requests did not complete " class="issue-link" data-issue-key="LU-12632"><del>LU-12632</del></a> mdt: wakeup HSM coordinator more oftenly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 45f978d85b91e0abe4d9983ad721b1c65f7eb820</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40387/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40387/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12632" title="sanity-hsm test_90: FAIL: requests did not complete " class="issue-link" data-issue-key="LU-12632"><del>LU-12632</del></a> hsm: wait longer in sanity-hsm test_90()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bde0ebd6a3e88de5b8d7681efe5d67b98e7fe6a0</p> <p>Landed for 2.14</p> Related LU-12870 Development Rank 1|i00kqn: Rank (Obsolete) 9223372036854775807 Severity