Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.6.0
-
3
-
12900
Description
Most archives fail (depends on MDT index). Successfully released files cannot be restored.
# export MOUNT_2=y
# export MDSCOUNT=4
# llmount.sh
# for ((i = 0; i < MDSCOUNT; i++)); do
mdt_name=$(printf 'lustre-MDT%04x' $i)
lctl conf_param $mdt_name.mdt.hsm_control=enabled
done
# mount $HOSTNAME@tcp:/lustre /mnt/lustre-hsm -t lustre -o user_xattr,flock
# lhsmtool_posix -vvvv --hsm_root=$HSM_ROOT --daemon /mnt/lustre-hsm 2> /tmp/hsm.log
# lfs mkdir -i0 -c4 /mnt/lustre/d0
# cd /mnt/lustre/d0
# echo > f0
# echo > f1
# echo > f2
# echo > f3
# lfs hsm_archive f0
# lfs hsm_state f0
f0: (0x00000001) exists, archive_id:1
# lfs hsm_release f0
Cannot send HSM request (use of f0): Operation not permitted
#
# lfs hsm_archive f1
# lfs hsm_archive f2
# lfs hsm_archive f3
# lfs hsm_release f1
# lfs hsm_release f2
Cannot send HSM request (use of f2): Operation not permitted
# lfs hsm_release f3
Cannot send HSM request (use of f3): Operation not permitted
# lfs hsm_state *
f0: (0x00000001) exists, archive_id:1
f1: (0x0000000d) released exists archived, archive_id:1
f2: (0x00000001) exists, archive_id:1
f3: (0x00000001) exists, archive_id:1
# cat f1
^C
Attachments
Activity
Please note that landing http://review.whamcloud.com/#/c/10866/ does not mean that this issue has been fixed.
I don't see any verification that restore either via lfs hsm_restore or by file access works as expected.
On current master I see that restore in a striped directory doesn't usually work.
Hmm, I tried to reproduce the problem, but seems works for me, though it seems the daemon can not be started correctly after the mount, and I have to start again.
[root@mds tests]# sh -vx tests.sh umount /mnt/lustre-hsm + umount /mnt/lustre-hsm MOUNT_2=y MDSCOUNT=4 sh llmount.sh + MOUNT_2=y + MDSCOUNT=4 + sh llmount.sh Stopping clients: mds /mnt/lustre (opts:) Stopping client mds /mnt/lustre opts: Stopping clients: mds /mnt/lustre2 (opts:) Stopping client mds /mnt/lustre2 opts: Stopping /mnt/mds1 (opts:-f) on mds Stopping /mnt/mds2 (opts:-f) on mds Stopping /mnt/mds3 (opts:-f) on mds Stopping /mnt/mds4 (opts:-f) on mds Stopping /mnt/ost1 (opts:-f) on mds Stopping /mnt/ost2 (opts:-f) on mds waited 0 for 31 ST ost OSS OSS_uuid 0 Loading modules from /work/lustre-release_new/lustre-release/lustre/tests/.. detected 8 online CPUs by sysfs libcfs will create CPU partition based on online CPUs debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super subsystem_debug=all -lnet -lnd -pinger gss/krb5 is not supported Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format mds2: /tmp/lustre-mdt2 Format mds3: /tmp/lustre-mdt3 Format mds4: /tmp/lustre-mdt4 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients mds environments Loading modules from /work/lustre-release_new/lustre-release/lustre/tests/.. detected 8 online CPUs by sysfs libcfs will create CPU partition based on online CPUs debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super subsystem_debug=all -lnet -lnd -pinger gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop /tmp/lustre-mdt1 /mnt/mds1 Started lustre-MDT0000 Starting mds2: -o loop /tmp/lustre-mdt2 /mnt/mds2 Started lustre-MDT0001 Starting mds3: -o loop /tmp/lustre-mdt3 /mnt/mds3 Started lustre-MDT0002 Starting mds4: -o loop /tmp/lustre-mdt4 /mnt/mds4 Started lustre-MDT0003 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 Started lustre-OST0001 Starting client: mds: -o user_xattr,flock mds@tcp:/lustre /mnt/lustre Starting client: mds: -o user_xattr,flock mds@tcp:/lustre /mnt/lustre2 Using TIMEOUT=20 disable quota as required /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0000.mdt.hsm_control=enabled + /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0000.mdt.hsm_control=enabled /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0001.mdt.hsm_control=enabled + /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0001.mdt.hsm_control=enabled /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0002.mdt.hsm_control=enabled + /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0002.mdt.hsm_control=enabled /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0003.mdt.hsm_control=enabled + /work/lustre-release_new/lustre-release/lustre/utils/lctl conf_param lustre-MDT0003.mdt.hsm_control=enabled mount -t lustre -o user_xattr,flock mds:/lustre /mnt/lustre-hsm + mount -t lustre -o user_xattr,flock mds:/lustre /mnt/lustre-hsm sleep 5 + sleep 5 /work/lustre-release_new/lustre-release/lustre/utils/lhsmtool_posix -vvvv --hsm_root=/mnt/lustre --daemon /mnt/lustre-hsm 2> /tmp/hsm.log + /work/lustre-release_new/lustre-release/lustre/utils/lhsmtool_posix -vvvv --hsm_root=/mnt/lustre --daemon /mnt/lustre-hsm /work/lustre-release_new/lustre-release/lustre/utils/lfs mkdir -i0 -c4 /mnt/lustre/d0 + /work/lustre-release_new/lustre-release/lustre/utils/lfs mkdir -i0 -c4 /mnt/lustre/d0 cd /mnt/lustre/d0 + cd /mnt/lustre/d0 echo > f0 + echo echo > f1 + echo echo > f2 + echo echo > f3 + echo [root@mds tests]# vim /tmp/hsm.log [root@mds tests]# sh -vx test_1.sh cd /mnt/lustre/d0 + cd /mnt/lustre/d0 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f0 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f0 sleep 2 + sleep 2 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f0 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f0 f0: (0x00000000) /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f0 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f0 Cannot send HSM request (use of f0): Operation not permitted /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f1 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f1 sleep 2 + sleep 2 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f1 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f1 f1: (0x00000000) /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f1 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f1 Cannot send HSM request (use of f1): Operation not permitted /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f2 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f2 sleep 2 + sleep 2 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f2 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f2 f2: (0x00000000) /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f2 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f2 Cannot send HSM request (use of f2): Operation not permitted /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f3 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f3 sleep 2 + sleep 2 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f3 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f3 f3: (0x00000000) /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f3 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f3 Cannot send HSM request (use of f3): Operation not permitted
It seems because the daemon is not being start up correctly, then I start the daemon again, it works
[root@mds tests]# /work/lustre-release_new/lustre-release/lustre/utils/lhsmtool_posix -vvvv --hsm_root=/mnt/lustre --daemon /mnt/lustre-hsm lhsmtool_posix[25558]: action=0 src=(null) dst=(null) mount_point=/mnt/lustre-hsm [root@mds tests]# lhsmtool_posix[25559]: waiting for message from kernel lhsmtool_posix[25559]: copytool fs=lustre archive#=1 item_count=1 lhsmtool_posix[25559]: waiting for message from kernel lhsmtool_posix[25560]: '[0x300000401:0x1:0x0]' action ARCHIVE reclen 72, cookie=0x5399b0c2 lhsmtool_posix[25560]: processing file 'd0/f3' lhsmtool_posix[25560]: archiving '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' lhsmtool_posix[25560]: saving stripe info of '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' in /mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp.lov lhsmtool_posix[25560]: going to copy data from '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' lhsmtool_posix[25560]: Going to copy 1 bytes /mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0 -> /mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp lhsmtool_posix[25560]: data archiving for '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' done lhsmtool_posix[25560]: attr file for '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' lhsmtool_posix[25560]: fsetxattr of 'trusted.hsm' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25560]: fsetxattr of 'trusted.link' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25560]: fsetxattr of 'trusted.lov' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25560]: fsetxattr of 'trusted.lma' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25560]: fsetxattr of 'lustre.lov' on '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25560]: xattr file for '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0_tmp' lhsmtool_posix[25560]: symlink '/mnt/lustre/shadow/d0/f3' to '../../0001/0000/0401/0000/0003/0000/0x300000401:0x1:0x0' done lhsmtool_posix[25560]: Action completed, notifying coordinator cookie=0x5399b0c2, FID=[0x300000401:0x1:0x0], hp_flags=0 err=0 lhsmtool_posix[25560]: llapi_hsm_action_end() on '/mnt/lustre-hsm/.lustre/fid/0x300000401:0x1:0x0' ok (rc=0) lhsmtool_posix[25559]: copytool fs=lustre archive#=1 item_count=1 lhsmtool_posix[25559]: waiting for message from kernel lhsmtool_posix[25562]: '[0x340000401:0x1:0x0]' action ARCHIVE reclen 72, cookie=0x5399b0c2 lhsmtool_posix[25562]: processing file 'd0/f0' lhsmtool_posix[25562]: archiving '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' lhsmtool_posix[25562]: saving stripe info of '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' in /mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp.lov lhsmtool_posix[25562]: going to copy data from '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' lhsmtool_posix[25562]: Going to copy 1 bytes /mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0 -> /mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp lhsmtool_posix[25562]: data archiving for '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' done lhsmtool_posix[25562]: attr file for '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' lhsmtool_posix[25562]: fsetxattr of 'trusted.hsm' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25562]: fsetxattr of 'trusted.link' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25562]: fsetxattr of 'trusted.lov' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25562]: fsetxattr of 'trusted.lma' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25562]: fsetxattr of 'lustre.lov' on '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25562]: xattr file for '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0_tmp' lhsmtool_posix[25562]: symlink '/mnt/lustre/shadow/d0/f0' to '../../0001/0000/0401/4000/0003/0000/0x340000401:0x1:0x0' done lhsmtool_posix[25562]: Action completed, notifying coordinator cookie=0x5399b0c2, FID=[0x340000401:0x1:0x0], hp_flags=0 err=0 lhsmtool_posix[25562]: llapi_hsm_action_end() on '/mnt/lustre-hsm/.lustre/fid/0x340000401:0x1:0x0' ok (rc=0) lhsmtool_posix[25559]: copytool fs=lustre archive#=1 item_count=1 lhsmtool_posix[25559]: waiting for message from kernel lhsmtool_posix[25563]: '[0x280000400:0x2:0x0]' action ARCHIVE reclen 72, cookie=0x5399b0c2 lhsmtool_posix[25563]: processing file 'd0/f1' lhsmtool_posix[25563]: archiving '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' to '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' lhsmtool_posix[25563]: saving stripe info of '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' in /mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp.lov lhsmtool_posix[25563]: going to copy data from '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' to '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' lhsmtool_posix[25563]: Going to copy 1 bytes /mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0 -> /mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp lhsmtool_posix[25563]: data archiving for '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' to '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' done lhsmtool_posix[25563]: attr file for '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' saved to archive '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' lhsmtool_posix[25563]: fsetxattr of 'trusted.hsm' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success) lhsmtool_posix[25563]: fsetxattr of 'trusted.link' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success) lhsmtool_posix[25563]: fsetxattr of 'trusted.lov' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success) lhsmtool_posix[25563]: fsetxattr of 'trusted.lma' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success) lhsmtool_posix[25563]: fsetxattr of 'lustre.lov' on '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' rc=0 (Success) lhsmtool_posix[25563]: xattr file for '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' saved to archive '/mnt/lustre/0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0_tmp' lhsmtool_posix[25563]: symlink '/mnt/lustre/shadow/d0/f1' to '../../0002/0000/0400/8000/0002/0000/0x280000400:0x2:0x0' done lhsmtool_posix[25563]: Action completed, notifying coordinator cookie=0x5399b0c2, FID=[0x280000400:0x2:0x0], hp_flags=0 err=0 lhsmtool_posix[25563]: llapi_hsm_action_end() on '/mnt/lustre-hsm/.lustre/fid/0x280000400:0x2:0x0' ok (rc=0) lhsmtool_posix[25559]: copytool fs=lustre archive#=1 item_count=1 lhsmtool_posix[25559]: waiting for message from kernel lhsmtool_posix[25564]: '[0x2c0000401:0x1:0x0]' action ARCHIVE reclen 72, cookie=0x5399b0c2 lhsmtool_posix[25564]: processing file 'd0/f2' lhsmtool_posix[25564]: archiving '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' lhsmtool_posix[25564]: saving stripe info of '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' in /mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp.lov lhsmtool_posix[25564]: going to copy data from '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' lhsmtool_posix[25564]: Going to copy 1 bytes /mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0 -> /mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp lhsmtool_posix[25564]: data archiving for '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' to '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' done lhsmtool_posix[25564]: attr file for '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' lhsmtool_posix[25564]: fsetxattr of 'trusted.hsm' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25564]: fsetxattr of 'trusted.link' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25564]: fsetxattr of 'trusted.lov' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25564]: fsetxattr of 'trusted.lma' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25564]: fsetxattr of 'lustre.lov' on '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' rc=0 (Success) lhsmtool_posix[25564]: xattr file for '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' saved to archive '/mnt/lustre/0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0_tmp' lhsmtool_posix[25564]: symlink '/mnt/lustre/shadow/d0/f2' to '../../0001/0000/0401/c000/0002/0000/0x2c0000401:0x1:0x0' done lhsmtool_posix[25564]: Action completed, notifying coordinator cookie=0x5399b0c2, FID=[0x2c0000401:0x1:0x0], hp_flags=0 err=0 lhsmtool_posix[25564]: llapi_hsm_action_end() on '/mnt/lustre-hsm/.lustre/fid/0x2c0000401:0x1:0x0' ok (rc=0) [root@mds tests]# sh -vx test_1.sh cd /mnt/lustre/d0 + cd /mnt/lustre/d0 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f0 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f0 sleep 2 + sleep 2 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f0 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f0 f0: (0x00000009) exists archived, archive_id:1 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f0 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f0 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f1 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f1 sleep 2 + sleep 2 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f1 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f1 f1: (0x00000009) exists archived, archive_id:1 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f1 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f1 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f2 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f2 sleep 2 + sleep 2 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f2 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f2 f2: (0x00000009) exists archived, archive_id:1 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f2 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f2 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f3 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_archive f3 sleep 2 + sleep 2 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f3 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_state f3 f3: (0x00000009) exists archived, archive_id:1 /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f3 + /work/lustre-release_new/lustre-release/lustre/utils/lfs hsm_release f3 [root@mds tests]#
So I suspect this is related with daemon, not about striped directory.
lfs hsm_state clearly shows that only f1 was successfully archived, that's the reason why files could not be released/restored.
Could you display what is in copytool log /tmp/hsm.log, MDTs logs and client logs?
Patch landed to master.