Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20972

lfs find: parallel find is OOM-killed on wide directories

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • Lustre 2.18.0
    • 3
    • 9223372036854775807

      Parallel find memory use grows with the number of directories that are found and not yet processed, without any limit. On a directory with many subdirectories, "lfs find" is killed by the OOM killer, where the serial traversal of the same tree needs a few MB. Memory use is about 75 KB for every queued directory. With "lfs find --xattr", copy_find_param() also copies some xattr name and value buffers for another 128 KB per queued directory.
      The work queue has no bound: a directory is queued as soon as its parent's readdir() returns it, and the parent's readdir() loop does not wait for the queue to drain. So a directory with N subdirectories puts N work units, i.e. N * 75 KB, on the queue at once: 1.5 GB for 20000 subdirectories, 7.5 GB for 100000.

      A quick reproducer, on a VM client with 1.4 GB of RAM and /mnt/lustre mounted:

      D=/mnt/lustre/wide; mkdir -p $D; cd $D
      seq -f "d%g" 1 20000 | xargs mkdir
      cd /
      lfs find --threads 1 $D | wc -l
      lfs find $D | wc -l
      

      Peak RSS read from VmHWM:

      lfs find --threads 1: rc=0   entries=20001 peakRSS=3068kB
      lfs find --threads 4: rc=137 entries=1749  peakRSS=888000kB
      lfs find:             rc=137 entries=1630  peakRSS=791256kB
      

      and in the kernel log:

      Out of memory: Killed process 8692 (lfs) total-vm:1283828kB, anon-rss:966400kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:2048kB oom_score_adj:0
      

      The same memory usage could be seen for every llapi_find_with_cb() user that sets fp_thread_count > 1, e.g. llapi_scan_namespace() and ltrash_purge.

            sebastien Sebastien Buisson
            sebastien Sebastien Buisson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: