[LU-15140] recovery-random-scale: No sub tests failed in this test set, FAIL: remove sub-test dirs failed

Related — Tue, 9 Jan 2024 17:45:07 +0000

This issue was created by maloo for Andreas Dilger <adilger@whamcloud.com>

This issue relates to the following test suite run:
https://testing.whamcloud.com/test_sets/dfbec373-e4f6-416f-83b9-2265475a3b80
https://testing.whamcloud.com/test_sets/0d2d3b02-4871-4160-a995-a51a74e4cd3b
https://testing.whamcloud.com/test_sets/f060ce64-9e1e-4546-b4bd-2f740787c589
https://testing.whamcloud.com/test_sets/2640a2ac-2fff-48d5-82a6-cab054add322
https://testing.whamcloud.com/test_sets/530acf01-f304-4dfa-94aa-16b5c1280cc5
https://testing.whamcloud.com/test_sets/902b1bb9-ed8b-400b-895a-baa328d4b7c5

== recovery-random-scale test complete, duration 85944 sec =========================================== 19:34:45 (1634672085)
rm: cannot remove '/mnt/lustre/d0.tar-trevis-68vm4.trevis.whamcloud.com/etc/selinux/targeted/active/modules/100': Directory not empty
 recovery-random-scale : @@@@@@ FAIL: remove sub-test dirs failed 
Stopping clients: trevis-68vm1.trevis.whamcloud.com,trevis-68vm3,trevis-68vm4 /mnt/lustre (opts:)
while umount  /mnt/lustre 2>&1 | grep -q busy; do
    echo /mnt/lustre is still busy, wait one second && sleep 1;
done;
fi
Stopping client trevis-68vm1.trevis.whamcloud.com /mnt/lustre opts:
Stopping client trevis-68vm3.trevis.whamcloud.com /mnt/lustre opts:
Stopping client trevis-68vm4.trevis.whamcloud.com /mnt/lustre opts:
COMMAND    PID USER   FD   TYPE      DEVICE SIZE/OFF               NODE NAME
run_tar.s 2934 root  cwd    DIR 1273,181606    11264 144116446786489609 /mnt/lustre/d0.tar-trevis-68vm4.trevis.whamcloud.com
tar       3222 root  cwd    DIR 1273,181606    11264 144116446786489609 /mnt/lustre/d0.tar-trevis-68vm4.trevis.whamcloud.com
tar       3223 root  cwd    DIR 1273,181606    11264 144116446786489609 /mnt/lustre/d0.tar-trevis-68vm4.trevis.whamcloud.com
tar       3223 root    3w   REG 1273,181606     5156 144117587637177621 /mnt/lustre/d0.tar-trevis-68vm4.trevis.whamcloud.com/etc/selinux/targeted/active/modules/100/gpg/cil
/mnt/lustre is still busy, wait one second
/mnt/lustre is still busy, wait one second
/mnt/lustre is still busy, wait one second
/mnt/lustre is still busy, wait one second
COMMAND    PID USER   FD      TYPE      DEVICE   SIZE/OFF               NODE NAME
run_dd.sh 2747 root  cwd   unknown 1273,181606                               /mnt/lustre/d0.dd-trevis-68vm3.trevis.whamcloud.com
dd        2772 root  cwd   unknown 1273,181606                               /mnt/lustre/d0.dd-trevis-68vm3.trevis.whamcloud.com
dd        2772 root    1w      REG 1273,181606 1160388608 144117486973878288 /mnt/lustre/d0.dd-trevis-68vm3.trevis.whamcloud.com/dd-file
/mnt/lustre is still busy, wait one second
/mnt/lustre is still busy, wait one second
/mnt/lustre is still busy, wait one second
/mnt/lustre is still busy, wait one second
:
:
Stopping clients: trevis-68vm1.trevis.whamcloud.com,trevis-68vm3,trevis-68vm4 /mnt/lustre2 (opts:)

so it looks like the client filesystem is eventually unmounted correctly, after the running jobs complete. It appears from the jobs that are still running that "tar" may still be writing into that directory tree at the time that "rm -r" is called, causing the directory not to be empty.

It would make sense to ensure that the running jobs are stopped before trying to delete the directory tree.

Whamcloud Community JIRA

[LU-15140] recovery-random-scale: No sub tests failed in this test set, FAIL: remove sub-test dirs failed