[LU-11709] sanity-hsm test 260c fails with 'request on 0x2000013a0:0x2ba:0x0 is not SUCCEED on mds1'

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.12.0
• Type: Bug
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: WC Triage
• Resolution: Duplicate
• Labels: DNE

Environment

DNE

• Severity: 3

Description

sanity-hsm test_260c fails to successfully archive a file and then starts the copytool. We see this test fail in review-dne-part-2 and review-dne-zfs-part-2 test groups.

Looking at logs for this failure at https://testing.whamcloud.com/test_sets/89b3068c-f258-11e8-b67f-52540065bddc , the last thing we see in the client test_log is waiting for the archive wait state to go from ARCHVIE to SUCCESS

Starting copytool agt1 on onyx-38vm2
CMD: onyx-38vm2 lhsmtool_posix  --daemon --hsm-root "/tmp/arc1/sanity-hsm.test_260c/" "/mnt/lustre2" < /dev/null > "/autotest/autotest/2018-11-27/lustre-reviews-el7_5-x86_64--review-dne-zfs-part-2--1_10_1__60277___2ab2a1f1-8a91-44cd-aca5-35ecd1cac72c/sanity-hsm.test_260c.copytool_log.onyx-38vm2.log" 2>&1
CMD: onyx-38vm2 mkdir -p /tmp/arc1/sanity-hsm.test_260c/
Starting copytool agt1 on onyx-38vm2
CMD: onyx-38vm2 lhsmtool_posix  --daemon --hsm-root "/tmp/arc1/sanity-hsm.test_260c/" --archive 2 "/mnt/lustre2" < /dev/null > "/autotest/autotest/2018-11-27/lustre-reviews-el7_5-x86_64--review-dne-zfs-part-2--1_10_1__60277___2ab2a1f1-8a91-44cd-aca5-35ecd1cac72c/sanity-hsm.test_260c.copytool2_log.onyx-38vm2.log" 2>&1
CMD: onyx-38vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x2000013a0:0x2ba:0x0'.*action='ARCHIVE'/ {print \$13}' | cut -f2 -d=
CMD: onyx-38vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x2000013a0:0x2ba:0x0'.*action='ARCHIVE'/ {print \$13}' | cut -f2 -d=
Waiting 200 secs for update
…
CMD: onyx-38vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x2000013a0:0x2ba:0x0'.*action='ARCHIVE'/ {print \$13}' | cut -f2 -d=
CMD: onyx-38vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x2000013a0:0x2ba:0x0'.*action='ARCHIVE'/ {print \$13}' | cut -f2 -d=
CMD: onyx-38vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x2000013a0:0x2ba:0x0'.*action='ARCHIVE'/ {print \$13}' | cut -f2 -d=
CMD: onyx-38vm4 /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x2000013a0:0x2ba:0x0'.*action='ARCHIVE'/ {print \$13}' | cut -f2 -d=
Update not seen after 200s: wanted 'SUCCEED' got 'WAITING'
 sanity-hsm test_260c: @@@@@@ FAIL: request on 0x2000013a0:0x2ba:0x0 is not SUCCEED on mds1 

There are no obvious issues found in the console nor in dmesg logs.

sanity-hsm test_206c just got updated with patch https://review.whamcloud.com/33478 that landed on November 26, 2018. This failure started on November 27, 2018.

Logs for other failures are at:
https://testing.whamcloud.com/test_sets/849ea658-f24c-11e8-86c0-52540065bddc
https://testing.whamcloud.com/test_sets/5714df60-f254-11e8-815b-52540065bddc

Issue Links

• is related to: LU-11572 tests: sanity-hsm test_260c is not re...
• is related to: LU-11653 hsm: copytool registration should tri...

Comments

[James Nunez (Inactive)]

Quentin - Would you please take a look at these failures? Either your patch brought out an existing issue with HSM or does not work well with DNE.

[James A Simmons]

Try the patch https://review.whamcloud.com/#/c/33649. Yep it was an out of order landing of HSM patches.

[Quentin Bouget]

My bad, I knew my patch for LU-11572 was not going to improve the situation with test_260c, but I didn't expect it to make things much worse.

James S. is right, https://review.whamcloud.com/#/c/33649 should solve this.

[Andreas Dilger]

The patch from LU-11572 has landed. Patches hitting this problem need to be rebased to get the fix.