[LU-5474] Test failure sanity-hsm test_90: requests did not complete

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Affects Version/s: Lustre 2.7.0, Lustre 2.5.4
• Labels: 22pl
• Severity: 3
• Assignee: James Nunez (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for Nathaniel Clark <[e-mail]>

This issue relates to the following test suite run:
https://testing.hpdd.intel.com/test_sets/90f1f8a2-1ff0-11e4-8610-5254006e85c2
https://testing.hpdd.intel.com/test_sets/29b71c62-020e-11e4-a47c-5254006e85c2
https://testing.hpdd.intel.com/test_sets/e775fca2-021b-11e4-9435-5254006e85c2

The sub-test test_90 failed with the following error:

requests did not complete

Info required for matching: sanity-hsm 90

Issue Links

• is related to: LU-3939 Test failure on test suite sanity-hsm, subtest test_40

Comments

[Peter Jones]

Landed for 2.7

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12069/
Subject: LU-5474 tests: sanity-hsm test_90 use local HSM_ARCHIVE
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8641822bdb248a034bf4065c7e2e60cdb7a47041

[James Nunez (Inactive)]

I've updated sanity-hsm test 90 to use a local disk for archive as was implemented in test 40. For the last two runs with this change, test 90 has failed to archive any files from the file list; https://testing.hpdd.intel.com/test_sessions/5ac85a16-6ee0-11e4-81e7-5254006e85c2 and https://testing.hpdd.intel.com/test_sets/616b778e-6b77-11e4-b1b4-5254006e85c2 .

For test 90, it seems that the archive number specified to the copytool is not the same number as the archive requests. The archive number specified in the call to the copytool is #2 and the archive requests are looking for archive #1. From the client test log:

CMD: shadow-3vm9 lhsmtool_posix  --daemon --hsm-root /tmp/d90.sanity-hsm --archive 2 --bandwidth 1 /mnt/lustre < /dev/null > /logdir/test_logs/2014-11-17/lustre-reviews-el6-x86_64--review-dne-part-2--2_9_1__28407__-70078147694660-145306/sanity-hsm.test_90.copytool2_log.shadow-3vm9.log 2>&1
...
Changed after 8s: from 'lrh=[type=10680000 len=136 idx=1/1846] fid=[0x400000401:0x1c4:0x0] dfid=[0x400000401:0x1c4:0x0] compound/cookie=0x546ab2b6/0x546ab2b6 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[]
lrh=[type=10680000 len=136 idx=1/1847] fid=[0x400000401:0x1c5:0x0] dfid=[0x400000401:0x1c5:0x0] compound/cookie=0x546ab2b6/0x546ab2b7 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[]

On the MDS, it complains that it can't find the agent, which is true, archive #2 was started by the copytool:

LustreError: 14430:0:(mdt_hsm_cdt_agent.c:338:mdt_hsm_agent_send()) lustre-MDT0000: Cannot find agent for archive 1: rc = -11

Yet, test 40 uses essentially the same calls to start the copytool and the archive numbers for this test are consistent. Test 90 archives from a file list and test 40 does not.

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: http://review.whamcloud.com/12069
Subject: LU-5474 tests: sanity-hsm test_90 use local HSM_ARCHIVE
Project: fs/lustre-release
Branch: master
Current Patch Set: 9
Commit: 7a0b4705f130f318acc0aaf6297cd4e0068060e5

[James Nunez (Inactive)]

Andreas,
I modified http://review.whamcloud.com/#/c/12069/ to use $TMP for the archive.

[Andreas Dilger]

Bruno, can you please make a patch to sanity-hsm test_90() to use $TMP for the HSM Archive, as was done with test_40()? Is there any reason not to make all HSM tests use $TMP for the archive by default, and only make some tests use the NFS shared directory if needed (very large archive, or HSM failover testing)? That would speed up all the HSM tests I think, unless it filled up $TMP and caused test failures.

[nasf (Inactive)]

Another failure instance on b2_5:
https://testing.hpdd.intel.com/test_sets/d03f6910-681e-11e4-acbe-5254006e85c2

[Bruno Faccini (Inactive)]

Andreas, yes this is tracked in LU-3939. And I remember that I already have made a similar+related comment about test_90 there ...

[Andreas Dilger]

I recall seeing a patch to move the HSM archive into $TMP instead of the shared directory. That might speed up the testing, as long as the archive doesn't need to be too large.

[James Nunez (Inactive)]

Test patch at http://review.whamcloud.com/#/c/12069/ .

This patch is to collect data on sanity-hsm test 90 failures. Looking over previous logs does not point to a problem in the test nor in the Lustre/HSM code. It just looks like archiving 51 files sometimes takes longer than 100 seconds. The patch tests if allowing for more time allows the files to be archived.

[Andreas Dilger]

It looks like the sanity-hsm test_90 exception was removed in the autotest framework, so it is possible to re-enable this test along with a patch to fix it.