[LU-4691] HSM operations mostly fail in striped directories

• Status: Closed
• Project: Lustre
• Affects Version/s: Lustre 2.6.0
• Fix Version/s: Lustre 2.7.0
• Type: Bug
• Priority: Critical
• Reporter: John Hammond
• Assignee: WC Triage
• Resolution: Fixed
• Labels: HSM, dne2, mdt
• Severity: 3

Description

Most archives fail (depends on MDT index). Successfully released files cannot be restored.

# export MOUNT_2=y
# export MDSCOUNT=4
# llmount.sh
# for ((i = 0; i < MDSCOUNT; i++)); do
    mdt_name=$(printf 'lustre-MDT%04x' $i)
    lctl conf_param $mdt_name.mdt.hsm_control=enabled
done
# mount $HOSTNAME@tcp:/lustre /mnt/lustre-hsm -t lustre -o user_xattr,flock
# lhsmtool_posix -vvvv --hsm_root=$HSM_ROOT --daemon /mnt/lustre-hsm 2> /tmp/hsm.log
# lfs mkdir -i0 -c4 /mnt/lustre/d0
# cd /mnt/lustre/d0
# echo > f0
# echo > f1
# echo > f2
# echo > f3
# lfs hsm_archive f0
# lfs hsm_state f0
f0: (0x00000001) exists, archive_id:1
# lfs hsm_release f0
Cannot send HSM request (use of f0): Operation not permitted
#  
# lfs hsm_archive f1
# lfs hsm_archive f2
# lfs hsm_archive f3
# lfs hsm_release f1
# lfs hsm_release f2
Cannot send HSM request (use of f2): Operation not permitted
# lfs hsm_release f3
Cannot send HSM request (use of f3): Operation not permitted
# lfs hsm_state *
f0: (0x00000001) exists, archive_id:1
f1: (0x0000000d) released exists archived, archive_id:1
f2: (0x00000001) exists, archive_id:1
f3: (0x00000001) exists, archive_id:1
# cat f1
^C

Comments

[Aurelien Degremont (Inactive)]

lfs hsm_state clearly shows that only f1 was successfully archived, that's the reason why files could not be released/restored.

Could you display what is in copytool log /tmp/hsm.log, MDTs logs and client logs?

[Di Wang]

Hmm, I tried to reproduce the problem, but seems works for me, though it seems the daemon can not be started correctly after the mount, and I have to start again.

[root@mds tests]# sh -vx tests.sh 
umount /mnt/lustre-hsm
+ umount /mnt/lustre-hsm
MOUNT_2=y  MDSCOUNT=4 sh llmount.sh
+ MOUNT_2=y
+ MDSCOUNT=4
+ sh llmount.sh
Stopping clients: mds /mnt/lustre (opts:)
Stopping client mds /mnt/lustre opts:
Stopping clients: mds /mnt/lustre2 (opts:)
Stopping client mds /mnt/lustre2 opts:
Stopping /mnt/mds1 (opts:-f) on mds
Stopping /mnt/mds2 (opts:-f) on mds
Stopping /mnt/mds3 (opts:-f) on mds
Stopping /mnt/mds4 (opts:-f) on mds
Stopping /mnt/ost1 (opts:-f) on mds
Stopping /mnt/ost2 (opts:-f) on mds
waited 0 for 31 ST ost OSS OSS_uuid 0
Loading modules from /work/lustre-release_new/lustre-release/lustre/tests/..
detected 8 online CPUs by sysfs
libcfs will create CPU partition based on online CPUs
debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super
subsystem_debug=all -lnet -lnd -pinger
gss/krb5 is not supported
Formatting mgs, mds, osts
Format mds1: /tmp/lustre-mdt1
Format mds2: /tmp/lustre-mdt2
Format mds3: /tmp/lustre-mdt3
Format mds4: /tmp/lustre-mdt4
Format ost1: /tmp/lustre-ost1
Format ost2: /tmp/lustre-ost2
Checking servers environments
Checking clients mds environments
Loading modules from /work/lustre-release_new/lustre-release/lustre/tests/..
detected 8 online CPUs by sysfs
libcfs will create CPU partition based on online CPUs
debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super
subsystem_debug=all -lnet -lnd -pinger
gss/krb5 is not supported
Setup mgs, mdt, osts
Starting mds1:   -o loop /tmp/lustre-mdt1 /mnt/mds1
Started lustre-MDT0000
Starting mds2:   -o loop /tmp/lustre-mdt2 /mnt/mds2
Started lustre-MDT0001
Starting mds3:   -o loop /tmp/lustre-mdt3 /mnt/mds3
Started lustre-MDT0002
Starting mds4:   -o loop /tmp/lustre-mdt4 /mnt/mds4
Started lustre-MDT0003
Starting ost1:   -o loop /tmp/lustre-ost1 /mnt/ost1
Started lustre-OST0000
Starting ost2:   -o loop /tmp/lustre-ost2 /mnt/ost2
Started lustre-OST0001
Starting client: mds: -o user_xattr,flock mds@tcp:/lustre /mnt/lustre
Starting client: mds: -o user_xattr,flock mds@tcp:/lustre /mnt/lustre2
Using TIMEOUT=20
disable quota as required
/work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0000.mdt.hsm_control=enabled
+ /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0000.mdt.hsm_control=enabled
/work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0001.mdt.hsm_control=enabled
+ /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0001.mdt.hsm_control=enabled
/work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0002.mdt.hsm_control=enabled
+ /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0002.mdt.hsm_control=enabled
/work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0003.mdt.hsm_control=enabled
+ /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0003.mdt.hsm_control=enabled
mount -t lustre -o user_xattr,flock mds:/lustre /mnt/lustre-hsm
+ mount -t lustre -o user_xattr,flock mds:/lustre /mnt/lustre-hsm
sleep 5
+ sleep 5
/work/lustre-release_new/lustre-release/lustre/utils/lhsmtool_posix -vvvv --hsm_root=/mnt/lustre --daemon /mnt/lustre-hsm 2> /tmp/hsm.log
+ /work/lustre-release_new/lustre-release/lustre/utils/lhsmtool_posix -vvvv --hsm_root=/mnt/lustre --daemon /mnt/lustre-hsm
/work/lustre-release_new/lustre-release/lustre/utils/lfs mkdir -i0 -c4 /mnt/lustre/d0
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs mkdir -i0 -c4 /mnt/lustre/d0
cd /mnt/lustre/d0
+ cd /mnt/lustre/d0
echo > f0
+ echo
echo > f1
+ echo
echo > f2
+ echo
echo > f3
+ echo

[root@mds tests]# vim /tmp/hsm.log 
[root@mds tests]# sh -vx test_1.sh 
cd /mnt/lustre/d0
+ cd /mnt/lustre/d0
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f0
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f0
sleep 2
+ sleep 2
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f0
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f0
f0: (0x00000000)
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f0
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f0
Cannot send HSM request (use of f0): Operation not permitted

/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f1
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f1
sleep 2
+ sleep 2
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f1
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f1
f1: (0x00000000)
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f1
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f1
Cannot send HSM request (use of f1): Operation not permitted

/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f2
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f2
sleep 2
+ sleep 2
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f2
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f2
f2: (0x00000000)
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f2
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f2
Cannot send HSM request (use of f2): Operation not permitted

/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f3
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f3
sleep 2
+ sleep 2
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f3
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f3
f3: (0x00000000)
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f3
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f3
Cannot send HSM request (use of f3): Operation not permitted

It seems because the daemon is not being start up correctly, then I start the daemon again, it works

[root@mds tests]# /work/lustre-release_new/lustre-release/lustre/utils/lhsmtool_posix -vvvv --hsm_root=/mnt/lustre --daemon /mnt/lustre-hsm 
lhsmtool_posix[25558]: action=0 src=(null) dst=(null) mount_point=/mnt/lustre-hsm
[root@mds tests]# lhsmtool_posix[25559]: waiting for message from kernel
lhsmtool_posix[25559]: copytool fs=lustre archive#=1 item_count=1
lhsmtool_posix[25559]: waiting for message from kernel
lhsmtool_posix[25560]: '[0x300000401:0x1:0x0]' action ARCHIVE reclen 72, cookie=0x5399b0c2
lhsmtool_posix[25560]: processing file 'd0/f3'
lhsmtool_posix[25560]: archiving '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp'
lhsmtool_posix[25560]: saving stripe info of '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' in /mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp.lov
lhsmtool_posix[25560]: going to copy data from '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp'
lhsmtool_posix[25560]: Going to copy 1 bytes /mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0 -> /mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp

lhsmtool_posix[25560]: data archiving for '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' done
lhsmtool_posix[25560]: attr file for '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp'
lhsmtool_posix[25560]: fsetxattr of 'trusted.hsm' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25560]: fsetxattr of 'trusted.link' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25560]: fsetxattr of 'trusted.lov' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25560]: fsetxattr of 'trusted.lma' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25560]: fsetxattr of 'lustre.lov' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25560]: xattr file for '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp'
lhsmtool_posix[25560]: symlink '/mnt/lustre/shadow/d0/f3' to '../../0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0' done
lhsmtool_posix[25560]: Action completed, notifying coordinator cookie=0x5399b0c2, FID=[0x300000401:0x1:0x0], hp_flags=0 err=0
lhsmtool_posix[25560]: llapi_hsm_action_end() on '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' ok (rc=0)
lhsmtool_posix[25559]: copytool fs=lustre archive#=1 item_count=1
lhsmtool_posix[25559]: waiting for message from kernel
lhsmtool_posix[25562]: '[0x340000401:0x1:0x0]' action ARCHIVE reclen 72, cookie=0x5399b0c2
lhsmtool_posix[25562]: processing file 'd0/f0'
lhsmtool_posix[25562]: archiving '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp'
lhsmtool_posix[25562]: saving stripe info of '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' in /mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp.lov
lhsmtool_posix[25562]: going to copy data from '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp'
lhsmtool_posix[25562]: Going to copy 1 bytes /mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0 -> /mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp

lhsmtool_posix[25562]: data archiving for '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' done
lhsmtool_posix[25562]: attr file for '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp'
lhsmtool_posix[25562]: fsetxattr of 'trusted.hsm' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25562]: fsetxattr of 'trusted.link' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25562]: fsetxattr of 'trusted.lov' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25562]: fsetxattr of 'trusted.lma' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25562]: fsetxattr of 'lustre.lov' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25562]: xattr file for '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp'
lhsmtool_posix[25562]: symlink '/mnt/lustre/shadow/d0/f0' to '../../0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0' done
lhsmtool_posix[25562]: Action completed, notifying coordinator cookie=0x5399b0c2, FID=[0x340000401:0x1:0x0], hp_flags=0 err=0
lhsmtool_posix[25562]: llapi_hsm_action_end() on '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' ok (rc=0)
lhsmtool_posix[25559]: copytool fs=lustre archive#=1 item_count=1
lhsmtool_posix[25559]: waiting for message from kernel
lhsmtool_posix[25563]: '[0x280000400:0x2:0x0]' action ARCHIVE reclen 72, cookie=0x5399b0c2
lhsmtool_posix[25563]: processing file 'd0/f1'
lhsmtool_posix[25563]: archiving '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' to '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp'
lhsmtool_posix[25563]: saving stripe info of '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' in /mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp.lov
lhsmtool_posix[25563]: going to copy data from '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' to '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp'
lhsmtool_posix[25563]: Going to copy 1 bytes /mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0 -> /mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp

lhsmtool_posix[25563]: data archiving for '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' to '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' done
lhsmtool_posix[25563]: attr file for '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' saved to archive '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp'
lhsmtool_posix[25563]: fsetxattr of 'trusted.hsm' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25563]: fsetxattr of 'trusted.link' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25563]: fsetxattr of 'trusted.lov' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25563]: fsetxattr of 'trusted.lma' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25563]: fsetxattr of 'lustre.lov' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25563]: xattr file for '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' saved to archive '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp'
lhsmtool_posix[25563]: symlink '/mnt/lustre/shadow/d0/f1' to '../../0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0' done
lhsmtool_posix[25563]: Action completed, notifying coordinator cookie=0x5399b0c2, FID=[0x280000400:0x2:0x0], hp_flags=0 err=0
lhsmtool_posix[25563]: llapi_hsm_action_end() on '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' ok (rc=0)
lhsmtool_posix[25559]: copytool fs=lustre archive#=1 item_count=1
lhsmtool_posix[25559]: waiting for message from kernel
lhsmtool_posix[25564]: '[0x2c0000401:0x1:0x0]' action ARCHIVE reclen 72, cookie=0x5399b0c2
lhsmtool_posix[25564]: processing file 'd0/f2'
lhsmtool_posix[25564]: archiving '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp'
lhsmtool_posix[25564]: saving stripe info of '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' in /mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp.lov
lhsmtool_posix[25564]: going to copy data from '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp'
lhsmtool_posix[25564]: Going to copy 1 bytes /mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0 -> /mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp

lhsmtool_posix[25564]: data archiving for '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' done
lhsmtool_posix[25564]: attr file for '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp'
lhsmtool_posix[25564]: fsetxattr of 'trusted.hsm' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25564]: fsetxattr of 'trusted.link' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25564]: fsetxattr of 'trusted.lov' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25564]: fsetxattr of 'trusted.lma' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25564]: fsetxattr of 'lustre.lov' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success)
lhsmtool_posix[25564]: xattr file for '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp'
lhsmtool_posix[25564]: symlink '/mnt/lustre/shadow/d0/f2' to '../../0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0' done
lhsmtool_posix[25564]: Action completed, notifying coordinator cookie=0x5399b0c2, FID=[0x2c0000401:0x1:0x0], hp_flags=0 err=0
lhsmtool_posix[25564]: llapi_hsm_action_end() on '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' ok (rc=0)

[root@mds tests]# sh -vx test_1.sh 
cd /mnt/lustre/d0
+ cd /mnt/lustre/d0
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f0
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f0
sleep 2
+ sleep 2
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f0
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f0
f0: (0x00000009) exists archived, archive_id:1
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f0
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f0

/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f1
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f1
sleep 2
+ sleep 2
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f1
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f1
f1: (0x00000009) exists archived, archive_id:1
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f1
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f1

/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f2
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f2
sleep 2
+ sleep 2
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f2
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f2
f2: (0x00000009) exists archived, archive_id:1
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f2
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f2

/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f3
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f3
sleep 2
+ sleep 2
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f3
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f3
f3: (0x00000009) exists archived, archive_id:1
/work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f3
+ /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f3

[root@mds tests]# 

So I suspect this is related with daemon, not about striped directory.

[John Hammond]

I don't see any verification that restore either via lfs hsm_restore or by file access works as expected.

On current master I see that restore in a striped directory doesn't usually work.

[Di Wang]

http://review.whamcloud.com/10866 add striped dir to sanity-hsm.sh

[John Hammond]

Please note that landing http://review.whamcloud.com/#/c/10866/ does not mean that this issue has been fixed.

[John Hammond]

Di's patch is at http://review.whamcloud.com/#/c/10866/.

[John Hammond]

Patch landed to master.