[LU-12632] sanity-hsm test_90: FAIL: requests did not complete

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.13.0, Lustre 2.12.3, Lustre 2.14.0, Lustre 2.12.4, Lustre 2.12.5
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Maloo

Description

This issue was created by maloo for jianyu <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/0c4bfdc4-b860-11e9-a1bd-52540065bddc

test_90 failed with the following error:

CMD: trevis-34vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED'
CMD: trevis-34vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED'
Update not seen after 100s: wanted '' got 'lrh=[type=10680000 len=136 idx=1/748] fid=[0x200001b71:0x229:0x0] dfid=[0x200001b71:0x229:0x0] compound/cookie=0x0/0x5d494bd8 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/749] fid=[0x200001b71:0x22a:0x0] dfid=[0x200001b71:0x22a:0x0] compound/cookie=0x0/0x5d494bd9 action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/750] fid=[0x200001b71:0x22b:0x0] dfid=[0x200001b71:0x22b:0x0] compound/cookie=0x0/0x5d494bda action=RESTORE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]'
 sanity-hsm test_90: @@@@@@ FAIL: requests did not complete 

<<Please provide additional information about the failure here>>

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity-hsm test_90 - requests did not complete

Issue Links

• is related to: LU-12870 sanity-hsm test 9A fails with “uuid Dumping not found in agent list on mds1”

Comments

[Hongchao Zhang]

there are two kinds of failures "requests did not complete".
1, On LDiskFS
the related HSM archive operations are not started, and it could be caused by the absence of "libtool"

CMD: onyx-34vm7 libtool --mode=e pkill -x lhsmtool_posix
onyx-34vm7: sh: libtool: command not found
CMD: onyx-34vm7 rm -rf /tmp/arc1/sanity-hsm.test_90/

it cause the previous copy tool can't be killed and affect the following copy tool.

2, On ZFS
some have the similar "libtool" issue like LDiskFS.
others were caused by the slow HSM Restore operations, it started to show from Jan 10th, 2019
https://testing.whamcloud.com/sub_tests/f4a00a5a-14fc-11e9-b7d4-52540065bddc (zfs 0.7.9, only 1 times)
https://testing.whamcloud.com/sub_tests/b4fe93f2-528d-11e9-a256-52540065bddc (zfs 0.7.12, only 1 times)
https://testing.whamcloud.com/sub_tests/e6a8aa78-70fb-11e9-a6f9-52540065bddc (zfs 0.7.13)
https://testing.whamcloud.com/sub_tests/bfa06a40-c91d-11e9-90ad-52540065bddc (zfs 0.8.1)

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36101
Subject: LU-12632 test: debug patch
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: faeb584eeb8a2d8c51dbe834f5d8c79ee3781783

[Hongchao Zhang]

As per the logs, the HSM_RESTORE operation was slow and didn't complete during 100 seconds,
but there is no obvious issues in the logs.

[Peter Jones]

Hongchao

Can you please investigate?

Thanks

Peter

[Emoly Liu]

+1 on master branch: https://testing.whamcloud.com/test_sets/b8aec0e0-c843-11e9-a25b-52540065bddc

[Jian Yu]

+1 on master branch: https://testing.whamcloud.com/test_sets/39afca24-c5ef-11e9-98c8-52540065bddc