[LU-4114] sanity-hsm test_90 failure: 'cannot release a file list'

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Minor
• Affects Version/s: Lustre 2.5.0
• Labels: HSM

Environment

Lustre 2.5.0-RC1, el6

OpenSFS cluster with combined MGS/MDS (c03), single OSS (c04) with two OSTs, archive MGS/MDS (c05), archive OST (c06) with two OSTs, archive OST2 (c07) with two OSTs, eight clients; one agent + client(c08), one robinhood/db + client(c09) and others just running as Lustre clients (c09, c10, c11, c12, c13,c14, c15)

• Severity: 3
• Assignee: WC Triage
• Reporter: James Nunez (Inactive)

Description

Test results are at: https://maloo.whamcloud.com/test_sessions/31dee29c-3687-11e3-8a71-52540035b04c

From the test log:

== sanity-hsm test 90: Archive/restore a file list == 09:49:07 (1381942147)
pdsh@c15: c08: ssh exited with exit code 1
Purging archive on c08
Starting copytool agt1 on c08
mdt.scratch-MDT0000.hsm_control=purge
Changed after 0s: from '' to 'lrh=[type=10680000 len=136 idx=1/1536] fid=[0x200000408:0x1d3:0x0] dfid=[0x200000408:0x1d3:0x0] compound/cookie=0x525ebaf5/0x525ebaf9 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1539] fid=[0x200000408:0x1d6:0x0] dfid=[0x200000408:0x1d6:0x0] compound/cookie=0x525ebaf5/0x525ebafc action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1540] fid=[0x200000408:0x1d7:0x0] dfid=[0x200000408:0x1d7:0x0] compound/cookie=0x525ebaf5/0x525ebafd action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1541] fid=[0x200000408:0x1d8:0x0] dfid=[0x200000408:0x1d8:0x0] compound/cookie=0x525ebaf5/0x525ebafe action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1542] fid=[0x200000408:0x1d9:0x0] dfid=[0x200000408:0x1d9:0x0] compound/cookie=0x525ebaf5/0x525ebaff action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1543] fid=[0x200000408:0x1da:0x0] dfid=[0x200000408:0x1da:0x0] compound/cookie=0x525ebaf5/0x525ebb00 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1544] fid=[0x200000408:0x1db:0x0] dfid=[0x200000408:0x1db:0x0] compound/cookie=0x525ebaf5/0x525ebb01 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1545] fid=[0x200000408:0x1dc:0x0] dfid=[0x200000408:0x1dc:0x0] compound/cookie=0x525ebaf5/0x525ebb02 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1546] fid=[0x200000408:0x1dd:0x0] dfid=[0x200000408:0x1dd:0x0] compound/cookie=0x525ebaf5/0x525ebb03 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1547] fid=[0x200000408:0x1de:0x0] dfid=[0x200000408:0x1de:0x0] compound/cookie=0x525ebaf5/0x525ebb04 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1548] fid=[0x200000408:0x1df:0x0] dfid=[0x200000408:0x1df:0x0] compound/cookie=0x525ebaf5/0x525ebb05 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1549] fid=[0x200000408:0x1e0:0x0] dfid=[0x200000408:0x1e0:0x0] compound/cookie=0x525ebaf5/0x525ebb06 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1551] fid=[0x200000408:0x1e2:0x0] dfid=[0x200000408:0x1e2:0x0] compound/cookie=0x525ebaf5/0x525ebb08 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1552] fid=[0x200000408:0x1e3:0x0] dfid=[0x200000408:0x1e3:0x0] compound/cookie=0x525ebaf5/0x525ebb09 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1553] fid=[0x200000408:0x1e4:0x0] dfid=[0x200000408:0x1e4:0x0] compound/cookie=0x525ebaf5/0x525ebb0a action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1554] fid=[0x200000408:0x1e5:0x0] dfid=[0x200000408:0x1e5:0x0] compound/cookie=0x525ebaf5/0x525ebb0b action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1556] fid=[0x200000408:0x1e7:0x0] dfid=[0x200000408:0x1e7:0x0] compound/cookie=0x525ebaf5/0x525ebb0d action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1557] fid=[0x200000408:0x1e8:0x0] dfid=[0x200000408:0x1e8:0x0] compound/cookie=0x525ebaf5/0x525ebb0e action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1558] fid=[0x200000408:0x1e9:0x0] dfid=[0x200000408:0x1e9:0x0] compound/cookie=0x525ebaf5/0x525ebb0f action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1559] fid=[0x200000408:0x1ea:0x0] dfid=[0x200000408:0x1ea:0x0] compound/cookie=0x525ebaf5/0x525ebb10 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1560] fid=[0x200000408:0x1eb:0x0] dfid=[0x200000408:0x1eb:0x0] compound/cookie=0x525ebaf5/0x525ebb11 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1561] fid=[0x200000408:0x1ec:0x0] dfid=[0x200000408:0x1ec:0x0] compound/cookie=0x525ebaf5/0x525ebb12 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1562] fid=[0x200000408:0x1ed:0x0] dfid=[0x200000408:0x1ed:0x0] compound/cookie=0x525ebaf5/0x525ebb13 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1563] fid=[0x200000408:0x1ee:0x0] dfid=[0x200000408:0x1ee:0x0] compound/cookie=0x525ebaf5/0x525ebb14 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1564] fid=[0x200000408:0x1ef:0x0] dfid=[0x200000408:0x1ef:0x0] compound/cookie=0x525ebaf5/0x525ebb15 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1565] fid=[0x200000408:0x1f0:0x0] dfid=[0x200000408:0x1f0:0x0] compound/cookie=0x525ebaf5/0x525ebb16 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1566] fid=[0x200000408:0x1f1:0x0] dfid=[0x200000408:0x1f1:0x0] compound/cookie=0x525ebaf5/0x525ebb17 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1567] fid=[0x200000408:0x1f2:0x0] dfid=[0x200000408:0x1f2:0x0] compound/cookie=0x525ebaf5/0x525ebb18 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1568] fid=[0x200000408:0x1f3:0x0] dfid=[0x200000408:0x1f3:0x0] compound/cookie=0x525ebaf5/0x525ebb19 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1569] fid=[0x200000408:0x1f4:0x0] dfid=[0x200000408:0x1f4:0x0] compound/cookie=0x525ebaf5/0x525ebb1a action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1570] fid=[0x200000408:0x1f5:0x0] dfid=[0x200000408:0x1f5:0x0] compound/cookie=0x525ebaf5/0x525ebb1b action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1571] fid=[0x200000408:0x1f6:0x0] dfid=[0x200000408:0x1f6:0x0] compound/cookie=0x525ebaf5/0x525ebb1c action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=1/1573] fid=[0x200000408:0x1f8:0x0] dfid=[0x200000408:0x1f8:0x0] compound/cookie=0x525ebaf5/0x525ebb1e action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]'
Waiting 100 secs for update
pdsh@c15: mds: ssh exited with exit code 1
Cannot send HSM request (use of /lustre/scratch/d0.sanity-hsm/d90/f.sanity-hsm.90.1): Device or resource busy
 sanity-hsm test_90: @@@@@@ FAIL: cannot release a file list 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:4264:error_noexit()
  = /usr/lib64/lustre/tests/test-framework.sh:4291:error()
  = /usr/lib64/lustre/tests/sanity-hsm.sh:2353:test_90()
  = /usr/lib64/lustre/tests/test-framework.sh:4530:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:4563:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:4433:run_test()
  = /usr/lib64/lustre/tests/sanity-hsm.sh:2359:main()
Dumping lctl log to /tmp/test_logs/2013-10-16/093701/sanity-hsm.test_90.*.1381942161.log

Issue Links

• is related to: LU-4275 sanity-hsm test_8 and many more failed: 36/91 passed