Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21025

quota: slave doesn't reintegrate after LWP reconnects to restarted master

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      When an MDT (the quota master) restarts, the LWP on each OST/MDT slave
      reconnects to it and goes through "lwp recover" (ptlrpc_connect_interpret(),
      OBD_CONNECT_LIGHTWEIGHT). lwp_import_event() then cancels the global quota
      locks locally on IMP_EVENT_INVALIDATE.

      qsd_glb_blocking_ast() doesn't restart reintegration for local cancels
      (LDLM_FL_LOCAL_ONLY), because it treats them as stack cleanup or eviction.
      qsd_conn_callback() is only called on the first connect. So nothing restarts
      reintegration: the slave's global and slave index copies stay out of date
      until the writeback thread notices and retries. In testing that took ~115
      seconds after the master restart.

      During that window the slave misses limits set on the master. An ID that
      wasn't enforced before isn't enforced on the OSTs, and its writes don't
      acquire quota from the master, so users can exceed new hard limits.

      Reproducer: sanity-quota test_7c (restart mds during reintegration). The test
      passes, but the OSTs need ~120s (wait_ost_reint) to reintegrate after the MDT
      restart. The debug log shows "lwp recover", then "losing global index lock"
      for each type, then no "Starting reintegration thread" until the writeback
      thread retries ~115s later.

      Measured (single-node VM, 1 MDT + 2 OSTs, master 2.17.60, 10 runs of
      test_7c each):

        without patch with patch
      total time of 10 runs 1377s 371s
      test_7c time, median (min-max) 140.5s (108-143) 24s (23-142)
      OST reintegration wait after MDT restart, median 120s 5s

      The single 142s run with the patch hit a separate race (see ticket 2). That
      race exists without this patch as well.

      Fix: restart reintegration on a local cancel of the global lock if the import
      is still valid (not on qsd/LWP cleanup or eviction), and make the
      reintegration thread wait for the import to be FULL before enqueueing the
      global lock.

            scherementsev Sergey Cheremencev
            scherementsev Sergey Cheremencev
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: