[LU-10677] can't delete directory

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.10.3
• Environment: zfs 0.7.5, OPA, skylake, centos7

Epic/Theme

dne

• Severity: 3
• Assignee: Lai Siyao
• Reporter: SC Admin

Description

Hiya,

we have 2 MDS's with 1 MDT on one of them and 2 MDTs on the other. so 3 MDT's in total. each MDT consists of 2 hardware raid1 mirrors with zmirror putting those together into one zfs MDT in one zpool. so 4-way replication.

latest centos7.4 kernels 3.10.0-693.17.1.el7.x86_64 everywhere. nopti set on lustre servers. 8 OSS's if that matters. multipath on all lustre servers. purely software raidz3 on OSS's.

we are testing DNE2 with 3-way dir striping, and also with default inheritance to all sub-dirs.
the below test fails and seems repeatable.

# lfs setdirstripe -c 3 mdt0-2
# lfs setdirstripe -D -c 3 mdt0-2
# chown rhumble mdt0-2
[rhumble@farnarkle2 ~]$ for f in /dagg/old_stuff/rjh/mdtest/mdt*; do echo === $f === ; time ( cd $f ; for g in {0000..9999}; do mkdir $g; for h in {00..99}; do mkdir $g/$h; done; done ) ; time rm -rf $f/*; done
...
=== /dagg/old_stuff/rjh/mdtest/mdt0-2 ===

real    57m21.053s
user    8m36.378s
sys     18m25.963s
rm: cannot remove ‘/dagg/old_stuff/rjh/mdtest/mdt0-2/2556’: Directory not empty

real    72m52.257s
user    0m4.197s
sys     7m59.024s
...

[rhumble@farnarkle2 ~]$ ls -al /dagg/old_stuff/rjh/mdtest/mdt0-2/2556
total 894
drwxrwxr-x 3 rhumble hpcadmin  76800 Feb 16 03:33 .
drwxr-xr-x 3 rhumble hpcadmin 838656 Feb 16 15:46 ..
[rhumble@farnarkle2 ~]$ rmdir /dagg/old_stuff/rjh/mdtest/mdt0-2/2556
rmdir: failed to remove ‘/dagg/old_stuff/rjh/mdtest/mdt0-2/2556’: Directory not empty
[rhumble@farnarkle2 ~]$ rm -rf /dagg/old_stuff/rjh/mdtest/mdt0-2/2556
rm: cannot remove ‘/dagg/old_stuff/rjh/mdtest/mdt0-2/2556’: Directory not empty

there aren't any problems seen with the other 4 dirs tested.
the other 4 dirs are mdt0, mdt1, mdt2 whcih have dir striping set to only that mdt and no default (-D) set, and to a directory with 3-way dir striping and no default (-D) set. ie.

[root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt0
lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none

[root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt1
lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none

[root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt2
lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none

[root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt0-2
lmv_stripe_count: 3 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64
mdtidx           FID[seq:oid:ver]
     0           [0x20000b7bd:0x4a1b:0x0]
     1           [0x28001639c:0x4a58:0x0]
     2           [0x680016b6b:0x4a58:0x0]

[root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt0-2-no-inherit
lmv_stripe_count: 3 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64
mdtidx           FID[seq:oid:ver]
     0           [0x20000bfa7:0xa63a:0x0]
     1           [0x2800182f7:0xa69f:0x0]
     2           [0x680018abd:0xa697:0x0]

[root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt0
lmv_stripe_count: 0 lmv_stripe_offset: -1 lmv_hash_type: none

[root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt1
lmv_stripe_count: 0 lmv_stripe_offset: -1 lmv_hash_type: none

[root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt2
lmv_stripe_count: 0 lmv_stripe_offset: -1 lmv_hash_type: none

[root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt0-2
lmv_stripe_count: 3 lmv_stripe_offset: -1 lmv_hash_type: fnv_1a_64

[root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt0-2-no-inherit/
lmv_stripe_count: 0 lmv_stripe_offset: -1 lmv_hash_type: none

the un-removable directories have only appeared on the 3-way -D directory, so I suspect the bug is to do with DNE2 and the -D inheritance stuff in particular.

I also re-ran the test with all 3 MDT's on one MDS, and the same thing happened - one directory was un-removable by any means.

there's nothing in dmesg or syslog.

cheers,
robin

Issue Links

• is related to: LU-8990 Failback LBUG lod_device_free()) ASSERTION( atomic_read(&lu->ld_ref)
• is related to: LU-10028 Export the information about Lustre special mount flags and options