Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21159

hybrid IO: buffered read()/write() fails with ENOMEM or EINVAL when the DIO bounce page pool is short

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • Lustre 2.17.0
    • None
    • master 8a35ae662e (2.17.61) client and servers, EL9.8 client (5.14.0-687.66.1.el9_8), x86_64, 4 GB RAM, ldiskfs OSTs; also 2.17.0
    • 3
    • 9223372036854775807

      An unaligned buffered read()/write() that hybrid IO switches to DIO fails when the bounce page pool cannot supply the whole sub-DIO, although buffered IO of the same call would succeed:

      • EINVAL when one sub-DIO needs more pages than the pool can ever hold (about 260 MiB on a 4 GB client). This is the np.save() failure in LU-19932.
      • ENOMEM when other processes hold the pool's pages. A few concurrent large unaligned IOs are enough.

      Reproducer

      Client with 4 GB RAM and 4 KiB pages, default tunables, ldiskfs OSTs:

      lfs setstripe -c 1 /mnt/lustre/d; cd /mnt/lustre/d
      # 1. one write at file offset 1, larger than the pool (pool_4k max_objects 66814 = 261 MiB here)
      dd if=/dev/zero of=f0 bs=300M count=1 seek=1 oflag=seek_bytes
      # 2. four writes at file offset 1 at the same time, each smaller than the pool
      for i in 1 2 3 4; do dd if=/dev/zero of=f$i bs=100M count=1 seek=1 oflag=seek_bytes status=none & done; wait
      # 3. any client: the existing fail_loc OBD_FAIL_LLITE_DIO_BUFFER_ALLOC, one 4 MiB write
      lctl set_param fail_loc=0x1438
      dd if=/dev/zero of=f5 bs=4M count=1 seek=1 oflag=seek_bytes
      lctl set_param fail_loc=0
      

      Output on master 8a35ae662e (client and servers 2.17.61, EL9.8 client, kernel 5.14.0-687.66.1.el9_8):

      # dd if=/dev/zero of=f0 bs=300M count=1 seek=1 oflag=seek_bytes
      dd: error writing 'f0': Invalid argument
      0 bytes copied, 0.0291946 s, 0.0 kB/s
      # for i in 1 2 3 4; do dd ... bs=100M count=1 seek=1 oflag=seek_bytes ... & done; wait
      dd: error writing 'f1': Cannot allocate memory
      dd: error writing 'f3': Cannot allocate memory
      # lctl get_param -n sptlrpc.page_pools   (pool_4k)
          max_objects: 66814
          total_objects: 66814
      # lctl set_param fail_loc=0x1438
      # dd if=/dev/zero of=f5 bs=4M count=1 seek=1 oflag=seek_bytes
      dd: error writing 'f5': Cannot allocate memory
      

      The same for reads: four concurrent 100 MiB dd reads at offset 1 gave 2 x "error reading: Cannot allocate memory", and an 8 MiB read at offset 1 with fail_loc=0x1438 also fails with ENOMEM. A 300 MiB os.write() from an unaligned Python buffer at file offset 0 fails with OSError: [Errno 22] Invalid argument.

      All three succeed with lctl set_param llite.*.hybrid_io=0, and with aligned writes (offset 0) with hybrid IO on. With oflag=direct the ENOMEM of case 3 is expected, the application asked for DIO.

      Analysis (master 8a35ae662e)

      1. ll_hybrid_bio_dio_switch_check() (lustre/llite/file.c:2114, thresholds at 2163-2173) switches the IO, do_file_write_iter() (file.c:2872) and do_file_read_iter() (file.c:2719) set IOCB_DIRECT.
      2. ll_direct_IO() (lustre/llite/rw26.c:333) sees an unaligned offset or iovec (rw26.c:365-368) and builds sub-DIOs of up to MAX_DIO_SIZE (rw26.c:473, about 576 MiB on x86_64). cl_dio_pages_init() calls ll_allocate_dio_buffer() (lustre/obdclass/cl_page.c:223, lustre/obdclass/cl_io.c:1491), which takes all bounce pages of the sub-DIO at once with obd_pool_get_folios_array() (cl_io.c:1524).
      3. The pool holds at most totalram_pages() / POOLS_COUNT pages (lustre/obdclass/page_pools.c:1049, 1079). __obd_pool_get_objects() returns -EINVAL for a request above that (page_pools.c:700), and -ENOMEM without waiting when the pool is at its maximum and nobody is growing it (page_pools.c:725-751). Not waiting is on purpose for ptlrpcd, but here the caller is the user process.
      4. The drain and retry of LU-19536 (rw26.c:506-542) retries only when earlier sub-DIOs of the same call are in flight (tot_bytes > bytes_at_drain, rw26.c:507-508). For the first sub-DIO, or when the pages are held by other processes, the error is returned through ll_file_io_generic() to write()/read(). -EINVAL is never retried.

      Related

      • LU-19932: the np.save() EINVAL on aarch64 (comment of 2026-08-14) is case 1.
      • LU-19536 (64351, drain and retry on ENOMEM): covers only later sub-DIOs of one call.
      • https://review.whamcloud.com/68541 (LU-19895, open): caps the bounce buffer at half the pool. With PS23 case 1 passes, cases 2 and 3 still fail.

      Suggested fix

      A hybrid-switched IO that gets -ENOMEM (or -EINVAL) from the bounce buffer allocation should be done as buffered IO, which is what the application asked for. Returning 0 from ll_direct_IO() (rw26.c:447) is not enough, since the switched DIO is lockless and took no extent lock for a buffered fallback inside it. A prototype redoes the IO from do_file_write_iter()/do_file_read_iter() through ll_file_io_generic() without IOCB_DIRECT, from the saved iterator and ki_pos (a failed DIO reports no bytes even when earlier stripe chunks were done), leaves out O_APPEND writes, and caps an unaligned sub-DIO at a fraction of the pool, as 68541 does, so that one call never exceeds the pool. With it all of the above pass, an O_DIRECT write still gets ENOMEM, and a new sanity test using fail_loc=0x1438 passes.

            wc-triage WC Triage
            markus.hilger Markus Hilger
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: