Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20589

llite: write(2) returns 0 when IO restart budget runs out

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      A buffered write(2) on a Lustre client can return 0 without an error
      while the data is dropped.

      ll_file_io_generic() restarts an IO when the lower layer sets
      ci_need_restart. The budget is RETRY_ATTEMPTS, 1000
      (lustre/llite/file.c:2058, :2081, :2241). One producer of the restart
      is a layout generation mismatch: can_populate_pages()
      (lustre/llite/vvp_io.c:76-92) compares the inode's cached layout
      generation against the generation the IO snapshotted, and
      vvp_io_write_start() then does RETURN(0) (lustre/llite/vvp_io.c:
      1461-1462 on 2.14.0-ddn259, :1262-1263 on master), a success that
      moves no bytes.

      If the generation keeps changing, all 1000 attempts are used. Control
      then falls through to RETURN(result > 0 ? result : rc) with result 0
      and rc 0. The io->ci_result fallback does not help, because
      cl_io_start() sets ci_result to 0 when the start method returns 0. The
      VFS gets 0 and write(2) returns 0.

      POSIX lets a write of a non-zero count return 0 only in narrow cases,
      and applications do not usually treat it as an error. glibc, MPI-IO
      and many application IO loops either stop or spin. The bytes are not
      written and nothing above D_VFSTRACE is logged, so the loss is silent.

      Exhausting the restart budget must be reported as an error, for
      example -EAGAIN or -EIO, with a rate-limited console message that
      names the FID and the reason for the restart. This is separate from
      the causes of the generation churn; on the reported system the churn
      came from client-side LDLM LRU reclaim under memory pressure.

      Seen at 2.14.0-ddn259. The llite files are byte-identical on
      b_es6_0, and master carries the same code.

      Related: LU-18435 removed one server-side cause of the churn (lod
      only, no client change). LU-6389 (gerrit 14123) added the restart loop
      and the 1000-attempt budget. Gerrit 67432 (LU-17936) adds a new
      -ENOLCK producer that reuses the same budget.

            hornc Chris Horn
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: