Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-10677

can't delete directory

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • Lustre 2.10.3
    • None
    • zfs 0.7.5, OPA, skylake, centos7
    • 3
    • 9223372036854775807

    Description

      Hiya,

      we have 2 MDS's with 1 MDT on one of them and 2 MDTs on the other. so 3 MDT's in total. each MDT consists of 2 hardware raid1 mirrors with zmirror putting those together into one zfs MDT in one zpool. so 4-way replication.

      latest centos7.4 kernels 3.10.0-693.17.1.el7.x86_64 everywhere. nopti set on lustre servers. 8 OSS's if that matters. multipath on all lustre servers. purely software raidz3 on OSS's.

      we are testing DNE2 with 3-way dir striping, and also with default inheritance to all sub-dirs.
      the below test fails and seems repeatable.

      # lfs setdirstripe -c 3 mdt0-2
      # lfs setdirstripe -D -c 3 mdt0-2
      # chown rhumble mdt0-2
      [rhumble@farnarkle2 ~]$ for f in /dagg/old_stuff/rjh/mdtest/mdt*; do echo === $f === ; time ( cd $f ; for g in {0000..9999}; do mkdir $g; for h in {00..99}; do mkdir $g/$h; done; done ) ; time rm -rf $f/*; done
      ...
      === /dagg/old_stuff/rjh/mdtest/mdt0-2 ===
      
      real    57m21.053s
      user    8m36.378s
      sys     18m25.963s
      rm: cannot remove ‘/dagg/old_stuff/rjh/mdtest/mdt0-2/2556’: Directory not empty
      
      real    72m52.257s
      user    0m4.197s
      sys     7m59.024s
      ...
      
      [rhumble@farnarkle2 ~]$ ls -al /dagg/old_stuff/rjh/mdtest/mdt0-2/2556
      total 894
      drwxrwxr-x 3 rhumble hpcadmin  76800 Feb 16 03:33 .
      drwxr-xr-x 3 rhumble hpcadmin 838656 Feb 16 15:46 ..
      [rhumble@farnarkle2 ~]$ rmdir /dagg/old_stuff/rjh/mdtest/mdt0-2/2556
      rmdir: failed to remove ‘/dagg/old_stuff/rjh/mdtest/mdt0-2/2556’: Directory not empty
      [rhumble@farnarkle2 ~]$ rm -rf /dagg/old_stuff/rjh/mdtest/mdt0-2/2556
      rm: cannot remove ‘/dagg/old_stuff/rjh/mdtest/mdt0-2/2556’: Directory not empty
      

      there aren't any problems seen with the other 4 dirs tested.
      the other 4 dirs are mdt0, mdt1, mdt2 whcih have dir striping set to only that mdt and no default (-D) set, and to a directory with 3-way dir striping and no default (-D) set. ie.

      [root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt0
      lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none
      
      [root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt1
      lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none
      
      [root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt2
      lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none
      
      [root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt0-2
      lmv_stripe_count: 3 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64
      mdtidx           FID[seq:oid:ver]
           0           [0x20000b7bd:0x4a1b:0x0]
           1           [0x28001639c:0x4a58:0x0]
           2           [0x680016b6b:0x4a58:0x0]
      
      [root@farnarkle1 ~]# lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt0-2-no-inherit
      lmv_stripe_count: 3 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64
      mdtidx           FID[seq:oid:ver]
           0           [0x20000bfa7:0xa63a:0x0]
           1           [0x2800182f7:0xa69f:0x0]
           2           [0x680018abd:0xa697:0x0]
      
      [root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt0
      lmv_stripe_count: 0 lmv_stripe_offset: -1 lmv_hash_type: none
      
      [root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt1
      lmv_stripe_count: 0 lmv_stripe_offset: -1 lmv_hash_type: none
      
      [root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt2
      lmv_stripe_count: 0 lmv_stripe_offset: -1 lmv_hash_type: none
      
      [root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt0-2
      lmv_stripe_count: 3 lmv_stripe_offset: -1 lmv_hash_type: fnv_1a_64
      
      [root@farnarkle1 ~]# lfs getdirstripe -D /dagg/old_stuff/rjh/mdtest/mdt0-2-no-inherit/
      lmv_stripe_count: 0 lmv_stripe_offset: -1 lmv_hash_type: none
      

      the un-removable directories have only appeared on the 3-way -D directory, so I suspect the bug is to do with DNE2 and the -D inheritance stuff in particular.

      I also re-ran the test with all 3 MDT's on one MDS, and the same thing happened - one directory was un-removable by any means.

      there's nothing in dmesg or syslog.

      cheers,
      robin

      Attachments

        1. 10677-warble2.log
          2.83 MB
        2. 10677-warble1.log
          21.52 MB
        3. 10677-rm58-warble2.log
          17.12 MB
        4. 10677-rm58-warble1.log
          39.61 MB

        Issue Links

          Activity

            People

              laisiyao Lai Siyao
              scadmin SC Admin
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated: