Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20624

utils: cb_get_dirstripe() reopen leaves the caller's directory fd stale

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Medium Medium
    • Lustre 2.18.0
    • None
    • None
    • 3
    • 9223372036854775807

      llapi_semantic_traverse() hands a directory descriptor to its sem_init
      callback by pointer, then calls fdopendir() on its own copy.
      cb_get_dirstripe() closes that descriptor and stores an O_NOFOLLOW reopen
      through the pointer when LL_IOC_LMV_GETSTRIPE answers ENOTTY, which it does
      for the fake symlink a foreign file or directory presents and for any
      directory not on Lustre at all.

      Two callers pass the address of a local copy, so the traversal keeps the
      number that was closed:

      lustre/utils/liblustreapi_pfind.c:2423   int d = dp == NULL ? -1 : *dp;
      lustre/utils/liblustreapi_pfind.c:2510       ret = cb_get_dirstripe(path, &d, param);
      
      lustre/utils/liblustreapi.c:3551         int d = dp == NULL ? -1 : *dp, fd = -1;
      lustre/utils/liblustreapi.c:3569             ret = cb_get_dirstripe(path, &d, param);
      

      It then descends on a closed descriptor, so every object below that
      directory is skipped in silence; closes that number a second time on the way
      out, by which point the walk may have reissued it to another thread; and
      leaks the descriptor that was opened.

      Reproduced on master at 5afbab284e with a tmpfs mounted inside a Lustre
      mount point, holding a.txt and sub/b.txt:

      # no predicate, so cb_get_dirstripe() is never called: correct
      $ lfs find /mnt/testfs/nl
      /mnt/testfs/nl
      /mnt/testfs/nl/a.txt
      /mnt/testfs/nl/sub
      /mnt/testfs/nl/sub/b.txt
      
      # any predicate needing the directory stripe: the subtree is gone
      $ lfs find /mnt/testfs/nl --printf '%p\n'
      lt-lfs: failed for '/mnt/testfs/nl': Operation not permitted
      $ echo $?
      1
      

      Each of these reproduces identically:

      --printf        --links        --mdt-count        --mdt-hash        --foreign
      

      The errno is wrong too: cb_get_dirstripe() returns the ioctl's -1 rather
      than a negative errno and cb_find_init() passes it out unchanged, so ENOTTY
      is reported as "Operation not permitted".

      lfs getdirstripe and lfs getstripe -D reach the same call through
      cb_getstripe(), but stop at the ENOTTY in any case, so there it is the
      double close that matters rather than a lost walk. Plain lfs getstripe -r
      sets neither fp_get_lmv nor fp_get_default_lmv and does not reach it.

      That makes lfs getdirstripe -r the clearest view of the descriptor, the two
      runs being identical until the final close (strace of the main process):

        openat(AT_FDCWD, "/mnt/testfs/nl2", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 3
        openat(AT_FDCWD, "/mnt/testfs/nl2", O_RDONLY|O_NONBLOCK|O_NOFOLLOW)  = 4
        close(3)                                = 0
        openat(AT_FDCWD, "/mnt/testfs/nl2", O_RDONLY|O_NONBLOCK|O_NOFOLLOW)  = 3
        close(4)                                = 0
        close(4)                                = -1 EBADF   <- without the fix
        close(3)                                = 0          <- with it
      

      3 is the traversal's descriptor and 4 the reopen. The third line is
      cb_get_dirstripe() closing the original, and the openat after it takes 3
      straight back while the caller still believes it owns that number.

      The fix is to write the descriptor back in both callers:

              ret = cb_get_dirstripe(path, &d, param);
              if (dp != NULL)
                      *dp = d;
      

      The third caller, in the dangling-symlink arm of cb_getstripe(), opens a
      descriptor it closes itself and is correct as it stands.

            hnishida Hiroshi Nishida
            hnishida Hiroshi Nishida
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: