[LU-15624] replay-single and ost-pools failed: rm: cannot remove 'd70b.replay-single': Directory not empty

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for paf <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/0877605b-4c45-4f18-b0f1-d70dc8853c88

replay-single and ost-pools fail in cleanup with the following message:

rm: cannot remove '/mnt/lustre/d70b.replay-single/onyx-70vm2.onyx.whamcloud.com': Directory not empty

Test 70b passed with no obvious errors.

Issue Links

• is duplicated by: LU-15395 replay-single: cannot remove '/mnt/lustre/d67b.replay-single': Directory not empty
• is related to: LU-10616 replay-single test_70b fails with 'rundbench load on <hostname(s)> failed!'
• is related to: LU-16336 LFSCK should fix inconsistencies caused by recovery abort

Comments

[Etienne Aujames]

+2 on master: https://testing.whamcloud.com/test_sets/e2e510ac-a17a-414e-b743-5c24c58a68d9
https://testing.whamcloud.com/test_sets/c3c3321e-6c32-4e21-90eb-0534253ae516

After replay-single, ost-pools tests fail with "mkdir: cannot create directory ...: No such file or directory":

00000020:00080000:1.0:1667480910.492873:0:463196:0:(update_trans.c:80:top_multiple_thandle_dump()) lustre-MDT0002-osd tmt 00000000d8f8efd0 refcount 3 committed 0 result -2 batchid 47244640363
00000020:00080000:1.0:1667480910.492874:0:463196:0:(update_trans.c:89:top_multiple_thandle_dump()) st 00000000c224f24a obd lustre-MDT0003-osp-MDT0002 committed 1 started 0 stopped 1 result -2 sub_th 0000000052ef663c
b4fe-99b3cdb2a5f5+9:630748:x1748469852443136:12345-10.240.26.136@tcp:36:mkdir.0 Request processed in 2079us (2126us total) trans 0 rc -2/-2

For example "ost-pools test_7a" has begun to fail since 2022-10-27 (https://testing.whamcloud.com/sub_tests/89ce4c69-2cee-49e1-9914-efbf713b5711):

$ git log --pretty="%h %ci %s" --since  2022-10-20 6aee406c84
b6b8fddf08b560acfcdf7c13c97e63                                                               
6aee406 2022-10-25 17:32:01 +0000 LU-14719 lod: distributed transaction check space
d4848d7 2022-10-25 17:27:57 +0000 LU-16187 tests: Fix is_project_quota_supported()
9309a32 2022-10-25 17:27:46 +0000 LU-16240 build: Use new AS_HELP_STRING
83d3f42 2022-10-25 17:27:36 +0000 LU-15305 obdclass: fix race in class_del_profile
1df5199 2022-10-25 17:27:27 +0000 LU-15807 ksocklnd: fix irq lock inversion while calling sk_
8a2bd96 2022-10-25 17:27:16 +0000 LU-16177 kernel: kernel update RHEL9.0 [5.14.0-70.26.1.el9_
5547158 2022-10-25 17:27:04 +0000 LU-16175 kernel: kernel update SLES12 SP5 [4.12.14-122.133.
5d31fda 2022-10-25 17:26:40 +0000 LU-16174 kernel: kernel update SLES15 SP4 [5.14.21-150400.2
c3467db 2022-10-25 17:26:19 +0000 LU-16173 kernel: kernel update SLES15 SP3 [5.3.18-150300.59
3493db6 2022-10-25 17:26:07 +0000 LU-16233 build: Add always target for SUSE15 SP3 LTSS
950e59c 2022-10-25 17:25:55 +0000 LU-15234 lnet: add mechanism for dumping lnd peer debug inf
0a639a3 2022-10-25 17:25:36 +0000 LU-15795 lbuild: enable KABI
c5a436d 2022-10-25 17:25:24 +0000 LU-12130 test: pool inheritance for mdt component
78dddb4 2022-10-25 17:25:11 +0000 LU-15447 tests: sanity-flr/208 reset rotational status
c24a38b 2022-10-25 17:25:03 +0000 LU-16183 test: sanity-hsm/70 should detect python
04e5fa7 2022-10-25 17:24:51 +0000 LU-13175 tests: sanity/803 to sync MDTs for actual statfs
2e08974 2022-10-25 17:24:37 +0000 LU-16139 statahead: avoid to block ptlrpcd interpret contex
a966b62 2022-10-25 17:24:24 +0000 LU-16149 lnet: Discovery queue and deletion race
0a317b1 2022-10-25 17:24:12 +0000 LU-15847 tgt: move tti_ transaction params to tsi_
4e2e8fd 2022-10-25 17:23:55 +0000 LU-15847 tgt: reply always with the latest assigned transno
4468f6c 2022-10-25 17:21:40 +0000 LU-16025 llite: adjust read count as file got truncated

[Andreas Dilger]

+1 on master: https://testing.whamcloud.com/test_sets/aede0a53-9f97-43e5-8451-823edddb23d9

[Andreas Dilger]

Also seen in sanity test_425:
https://testing.whamcloud.com/test_sets/e73a2124-550c-4145-a7fa-496e260b8d5d

== sanity test 425: lock count should not exceed lru size ========================================================== 21:18:05 (1651180685)
striped dir -i1 -c4 -H crush /mnt/lustre/d425.sanity
[e-mail]_size=100
ldlm.namespaces.lustre-MDT0000-mdc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-MDT0001-mdc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-MDT0002-mdc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-MDT0003-mdc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-OST0000-osc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-OST0001-osc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-OST0002-osc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-OST0003-osc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-OST0004-osc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-OST0005-osc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-OST0006-osc-ffff930cda456800.lru_size=100
ldlm.namespaces.lustre-OST0007-osc-ffff930cda456800.lru_size=100
rm: cannot remove '/mnt/lustre/d425.sanity': Directory not empty