[LU-12632] sanity-hsm test_90: FAIL: requests did not complete

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.13.0, Lustre 2.12.3, Lustre 2.14.0, Lustre 2.12.4, Lustre 2.12.5
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Maloo

Description

This issue was created by maloo for jianyu <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/0c4bfdc4-b860-11e9-a1bd-52540065bddc

test_90 failed with the following error:

CMD: trevis-34vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED'
CMD: trevis-34vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED'
Update not seen after 100s: wanted '' got 'lrh=[type=10680000 len=136 idx=1/748] fid=[0x200001b71:0x229:0x0] dfid=[0x200001b71:0x229:0x0] compound/cookie=0x0/0x5d494bd8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/749] fid=[0x200001b71:0x22a:0x0] dfid=[0x200001b71:0x22a:0x0] compound/cookie=0x0/0x5d494bd9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/750] fid=[0x200001b71:0x22b:0x0] dfid=[0x200001b71:0x22b:0x0] compound/cookie=0x0/0x5d494bda action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]'
 sanity-hsm test_90: @@@@@@ FAIL: requests did not complete 

<<Please provide additional information about the failure here>>

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity-hsm test_90 - requests did not complete

Issue Links

• is related to: LU-12870 sanity-hsm test 9A fails with “uuid Dumping not found in agent list on mds1”

Comments

[Chris Horn]

+1 on master https://testing.whamcloud.com/test_sets/8f0417bf-f011-4a15-a624-3c06ff83bc1d

[Nikitas Angelinas]

+1 on master https://testing.whamcloud.com/test_sets/070d81f7-e5a2-4fa3-8b8c-67a7820e29be

[Nikitas Angelinas]

+1 on master https://testing.whamcloud.com/test_sets/5ce24eb6-6fb9-404c-8bb0-0c87e9f2cede

[Emoly Liu]

more on master: 

https://testing.whamcloud.com/test_sets/7d2382e8-70e4-4b1f-b455-0e9c9b1e1d1b

https://testing.whamcloud.com/test_sets/404f16d2-3b8a-4bd2-8339-5142f548b657

[Jian Yu]

Still failed on master branch: https://testing.whamcloud.com/test_sets/13b3b0f2-406e-11ea-ac52-52540065bddc

[Jian Yu]

+1 on master branch: https://testing.whamcloud.com/test_sets/016d71b8-3c30-11ea-b1e8-52540065bddc

[Emoly Liu]

+1 on master: https://testing.whamcloud.com/test_sets/a70981de-0fb2-11ea-bbc3-52540065bddc

[Li Xi]

Example on master branch: https://testing.whamcloud.com/test_sets/d52265ac-d409-11e9-9fc9-52540065bddc

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36152
Subject: LU-12632 tests: stop running sanity-hsm 90 for ZFS
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5df30a16128b23977ef92de3ca06ec59a5f531de

[James A Simmons]

How? Its in the spec and deb files for BuildRequires and Requires for the test. How are the images being constructed in the Maloo environment?

[Hongchao Zhang]

there are two kinds of failures "requests did not complete".
1, On LDiskFS
the related HSM archive operations are not started, and it could be caused by the absence of "libtool"

CMD: onyx-34vm7 libtool --mode=e pkill -x lhsmtool_posix
onyx-34vm7: sh: libtool: command not found
CMD: onyx-34vm7 rm -rf /tmp/arc1/sanity-hsm.test_90/

it cause the previous copy tool can't be killed and affect the following copy tool.

2, On ZFS
some have the similar "libtool" issue like LDiskFS.
others were caused by the slow HSM Restore operations, it started to show from Jan 10th, 2019
https://testing.whamcloud.com/sub_tests/f4a00a5a-14fc-11e9-b7d4-52540065bddc (zfs 0.7.9, only 1 times)
https://testing.whamcloud.com/sub_tests/b4fe93f2-528d-11e9-a256-52540065bddc (zfs 0.7.12, only 1 times)
https://testing.whamcloud.com/sub_tests/e6a8aa78-70fb-11e9-a6f9-52540065bddc (zfs 0.7.13)
https://testing.whamcloud.com/sub_tests/bfa06a40-c91d-11e9-90ad-52540065bddc (zfs 0.8.1)